Home/ Questions/Q 8536277
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-11T10:37:30+00:00

I would like to parse text files (stored locally) with lxml’s etree. But all

  • 0

I would like to parse text files (stored locally) with lxml’s etree. But all of my files (thousands) have headers, such as:

-----BEGIN PRIVACY-ENHANCED MESSAGE-----
Proc-Type: 2001,MIC-CLEAR
Originator-Name: webmaster@www.sec.gov
Originator-Key-Asymmetric:
 MFgwCgYEVQgBAQICAf8DSgAwRwJAW2sNKK9AVtBzYZmr6aGjlWyK3XmZv3dTINen
 TWSM7vrzLADbmYQaionwg5sDW3P6oaM5D3tdezXMm7z1T+B+twIDAQAB
MIC-Info: RSA-MD5,RSA,
 AHxm/u6lqdt8X6gebNqy9afC2kLXg+GVIOlG/Vrrw/dTCPGwM15+hT6AZMfDSvFZ
 YVPEaPjyiqB4rV/GS2lj6A==

<SEC-DOCUMENT>0001193125-07-200376.txt : 20070913
<SEC-HEADER>0001193125-07-200376.hdr.sgml : 20070913
<ACCEPTANCE-DATETIME>20070913115715
ACCESSION NUMBER:       0001193125-07-200376
CONFORMED SUBMISSION TYPE:  10-K
PUBLIC DOCUMENT COUNT:      7
CONFORMED PERIOD OF REPORT: 20070630
FILED AS OF DATE:       20070913
DATE AS OF CHANGE:      20070913

and the first < isn’t until line 51 in this case (and isn’t 51 in all cases). The xml portions starts as follows:

</SEC-HEADER>
<DOCUMENT>
<TYPE>10-K
<SEQUENCE>1
<FILENAME>d10k.htm
<DESCRIPTION>FORM 10-K
<TEXT>
<HTML><HEAD>
<TITLE>Form 10-K</TITLE>
</HEAD>
 <BODY BGCOLOR="WHITE">
<h5 align="left"><a href="#toc">Table of Contents</a></h5>

Can I handle this on-the-fly with lxml? Or should I use a stream editor to omit each file’s header? Thanks!

Here is my current code and error.

from lxml import etree
f = etree.parse('temp.txt')

XMLSyntaxError: Start tag expected, '<' not found, line 1, column 1

Edit:

FWIW, here is a link to the file.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Given that there’s a standard for these files, it’s possible to write a proper parser rather than guessing at things, or hoping beautifulsoup gets things right. That doesn’t mean it’s the best answer for you, but it’s certainly work looking at.

    According to the standard at http://www.sec.gov/info/edgar/pdsdissemspec910.pdf what you’ve got (inside the PEM enclosure) is an SGML document defined by the provided DTD. So, first go to pages 48-55, extract the text there, and save it as, say, “edgar.dtd”.

    The first thing I’d do is install SP and use its tools to make sure that the documents really are valid and parseable by that DTD, to make sure you don’t waste a bunch of time on something that isn’t going to pan out.

    Python comes with a validating SGML parser, sgmllib. Unfortunately, it was never quite finished, and it’s deprecated in 2.6-2.7 (and removed in 3.x). But that doesn’t mean it won’t work. So, try it and see if it works.

    If not, I don’t know of any good alternatives in Python; most of the SGML code out there is in C, C++, or Perl. But you can wrap up any C or C++ library (I’d start with SP) pretty easily, as long as you’re comfortable writing your own wrapped in C/Cython/boost-python/whatever or using ctypes. You only need to wrap up the top-level functions, not build a complete set of bindings. But if you’ve never done anything like this before, it’s probably not the best time to learn.

    Alternatively, you can wrap up a command-line tool. SP comes with nsgmls. There’s another good tool written in perl with the same name (I think part of http://savannah.nongnu.org/projects/perlsgml/ but I’m not positive.) And dozens of other tools.

    Or, of course, you could write the whole thing, or just the parsing layer, in perl (or C++) instead of Python.

    • 0