[Semibug] Piping, redirection and shellscipts: 3/5/2025 7pm Eastern Standard time

[Steve Litt]

Ron said on Mon, 3 Mar 2025 02:42:50 -0800

>Steve Litt wrote on 2025-03-02 15:03:
>
>> Yesterday I was tasked with getting every domain name I owned into
>> my list at 444domains.com/domains , which is created by a
>> shellscript and Python program that read a Yaml file and convert it
>> to the web page. So I had to do the following:
>> 
>> * Screen scrape all my domain names from ionos, put in file
>> 
>> * Screen scrape all my domain names from Spaceship.Com, put in
>> another file
>> 
>> * Use Vim to remove everything but the domains themselves
>> 
>> * Sort both files
>> 
>> * Create a Python program to sort my Yaml file
>> 
>> * Run diff between each file and the sorted domains of the Yaml file 
>> - The output with the left arrows need to be added to Yaml file
>> 
>> * Re-sort the Yaml file
>> 
>> * Run the shellscript to create the web page.
>> 
>> This sounds straightforward  
>
>No, it does *not* sound straightforward!
>
>
>Is it just me, or does it sound like a Rube Goldberg contraption in
>software?

Systemd is a Rube Goldberg contraption, what I did is a series of
trivial operations. I didn't even bother to put them in a shellscript
because it's a one time operation.

>The list of all domain info should be in a DB or at least a text file,
>for starters.

A Yaml file is human readable text.

> From there, do the sort(s), build the YAML, etc. Why sort a YAML
> file?!?

Because when you do a merge, the two lists you're merging must be
sorted. Both of them. Unless you want to loop through one, and for each
loop through the other, keep track of the lowest, never mind, it would
be a mess.

>Running `diff` and manually merging?

I ran diff to get a list of domains in the registrar that aren't in the
yaml, and, wrote a 7 line shellscript to turn them into Yaml records
with each domain as the value of domain: , filled them in with sales
price, comment, when they were bought, price I paid, and price for
renewal. Then I ran a tiny Python program to once again sort the Yaml
file.

>
>
>All this scraping seems pointless, why is the source of this data
>inside web pages?

Because the registrars don't give you a nice JSON view of all your
domains: They give them to you in a web page table. So you need to copy
the web page, paste it into a file, and then massage the file to be
Yaml. Not difficult, especially if you use Vim or another full featured
editor.

The whole thing took about 90 minutes, for a process I had never before
done or thought about. This is the power of Unix. What alternative would
be more straightforward?

SteveT

Steve Litt 

http://444domains.com