Member-only story
Programming
Why Isn’t This Tool Part of Every Operating System?
What am I missing here?
4 min readJun 10, 2021
Years back I sometimes needed to scrape data from website forms. I’d write simple Perl code that looked something like this:
#!/usr/bin/perl
use HTTP::Request::Common qw(POST);
use LWP::UserAgent;$var1=”2012/08/01";
$button=”Send”;$ua=LWP::UserAgent->new();
my $req= POST ‘https://somesite/formpost.pl', [var1=> $var1,button=>$button];
$content=$ua->request($req)->as_string;
print $content;
You don’t need to understand that; what’s important is that there often wasn’t much more than that to write. Simple code for a simple task.
When I had the data, another program massaged it and perhaps plugged it in to yet another program which generated a web page or a report. Again, pretty simple stuff, made even easier because these were Unix or Linux servers. I’d use “expect” to automate all the massaging and plugging.
