Archives for posts with tag: annual report

Part 1 and Part 2 show the history and evolution of my attempts to web scrape EDGAR files from the SEC. You can see detailed discussions of the approaches in the previous posts. The implementation issues with Part 2 caused me to rethink my approach to the problem.

I was hesitant to use XBRL tags initially because 1) this data isn’t currently audited and 2) my dissertation used both XBRL data and hand-collected data and I saw differences around 30% of the time between what I hand-collected vs. the XBRL tags. But my opinion now is that using the XBRL tags is the only viable solution to the problem. Previously, I was able to scrape the actual titles used by the company for each financial statement line item, but the substantial amount of variation in the titles used for the same accounts made appending this data together into one DataFrame problematic (e.g., slight differences in naming conventions cause the same underlying variable to be spread out among multiple columns for different firms. This could be coded around in theory, but in practice it would be a nightmare).

Read the rest of this entry »

An automated approach to web scraping publicly traded companies’ financial statements is something that I’ve been working on for a while. My first post¬†identified the balance sheet due to a firm-specific characteristic of FCCY’s 12/31/2017 balance sheet- namely that it had a row titled “ASSETS.” Of course, not every firm is going to have this header in all caps to identify which table in the .html file is the balance sheet. But it was a start. The next, currently unpublished, step pulled the annual report and then identified the balance sheet using the summation of the number of times accounts commonly found on balance sheets were present.

Some of the issues I ran into while programming this: ¬†the amount of account variation between balance sheets of companies in different industries, the presence of extra spaces or characters present in the .html file that are not readily apparent to human eyes, the use of capital letters in account titles (or not) by different firms, and variation in how the same account may be called by different companies (e.g., additional paid in capital vs. additional paid-in capital and stockholder’s equity(deficiency) vs. shareholders’ equity vs. stockholders’ equity, etc.), financial statements being split in two across separate pages and identified in the file as two separate tables, notes at the bottom of the page with the financial statements also being tagged as tables by the issuer, substantial variation in the exact titles firms use for the various financial statements, and the actual layout of the tables after they have been scraped (e.g., multiple columns for a given year with data spread across the columns).

All of these are things that can be programmed around, and some of these issues we will see later in the post with FCCY’s 12/31/2017 10-K after we scrape it.

Read the rest of this entry »