[omaha] Omaha Digest, Vol 131, Issue 4

[Charlie Ahern]

Hey, Burch. Thanks for the reply. I'll have to do some research about the
SEC's HTTP access and alternatives to scraping. I'm still an ignorant
newbie.

Hopefully we can meet up at the Meetup.

Thanks again for your comments.

Charlie

On Tue, Apr 24, 2018 at 11:00 AM, <omaha-request at python.org> wrote:

> Send Omaha mailing list submissions to
>         omaha at python.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://mail.python.org/mailman/listinfo/omaha
> or, via email, send a message with subject or body 'help' to
>         omaha-request at python.org
>
> You can reach the person managing the list at
>         omaha-owner at python.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Omaha digest..."
>
>
> Today's Topics:
>
>    1. EDGAR Access (Burch Kealey)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 23 Apr 2018 19:57:35 +0000
> From: Burch Kealey <bkealey at unomaha.edu>
> To: Omaha Python Users Group <omaha at python.org>
> Subject: [omaha] EDGAR Access
> Message-ID:
>         <CO2PR07MB2645959722A42B312FC8B73CAC890 at CO2PR07MB2645.
> namprd07.prod.outlook.com>
>
> Content-Type: text/plain; charset="iso-8859-1"
>
> I spent a very limited amount of time trying to understand what Charlie
> meant by his comment that the SEC 'pandas-datareader feed'.  Is this just
> because the SEC shut down ftp access?  Frankly the fact that the shut down
> FTP access is probably a good thing - I am not sure of all the technical
> details of how they provide computing resources to FTP versus HTTP but our
> experience is that HTTP is is so much faster than the old FTP access.
> Further, at the peak the SEC only allowed 50 concurrent FTP sessions so
> there were plenty of times when we would try to establish a session and
> would be turned away because all of the ports were taken.  We have never
> had a competitive connection problem after going to HTTP which was about
> two years before they switched off the FTP.
>
>
> I will also observe that to me the frustrating issue about the SEC is how
> they make more data available for free.  You might like that but when they
> originally sought funding for EDGAR what they told Congress is we will
> build a repository and then let the market develop  unique products from
> that data.  So we have this constant risk that the SEC will decide to make
> something that we do freely available and that puts a lid on investment
> opportunities and other issues.  I would be interested in trying to better
> understand the issue Charlie raised because that is pretty much contrary to
> the way the SEC has worked.  They have had/provided a feed of EDGAR since
> EDGAR was first created so I don't think that can be what you are talking
> about.  One of the problems their contractor has had is that people can
> very easily develop tools to scrape filings so they have not gotten the
> revenue that they have expected from providing the direct feed.
>
>
> I know some folks at the SEC and have had at least a working relationship
> with them for almost a decade.  They were often frustrated because of users
> failing to observe the limitations for fair access that they tried to set
> out there was a ton of abuse.  They are always looking for ways to improve
> free access to content.
>
>
> We use the SECs RSS feed to identify when new filings are made and then
> REQUESTS to grab the filing for our internal use.  It is considerably
> faster than FTP access and we have far fewer problems in transit.  I don't
> understand why but that is my experience as someone who consumes 1,500 +
> SEC filings every single business day.
>
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Omaha mailing list
> Omaha at python.org
> https://mail.python.org/mailman/listinfo/omaha
>
>
> ------------------------------
>
> End of Omaha Digest, Vol 131, Issue 4
> *************************************
>



-- 
Charlie Ahern
 (408) 821-6414

Reading, after a certain age, diverts the mind too much from its creative
pursuits. Any man who reads too much and uses his own brain too little
falls into lazy habits of thinking, just as the man who spends too much
time in the theater is tempted to be content with living vicariously
instead of living his own life. - Albert Einstein