Jsonify: XML to JSON

pygixml.jsonify serializes XML directly to JSON. “Directly” is the operative word: the in-memory entry points (dumps() and friends) traverse the pugixml DOM in C++ and write straight into a JSON string buffer: no intermediate Python dict/list is ever built, unlike going through pygixml.dictify.parse() followed by json.dumps(). The streaming entry points (stream_dump(), stream_jsonl()) go a step further and skip even the DOM, converting a giant XML file to a giant JSON file in roughly constant memory.

The output shape matches pygixml.dictify.parse() exactly (same @-prefixed attributes, #text for mixed content, repeated siblings collapsed into arrays): jsonify.dumps(xml) is equivalent to, but faster than, json.dumps(dictify.parse(xml)).

from pygixml import jsonify

xml = """
<database name="users_db">
    <user id="101">
        <name>Mohammad</name>
        <balance>450.75</balance>
    </user>
    <tag>active</tag>
    <tag>verified</tag>
</database>
"""

jsonify.dumps(xml)
# '{"database": {"@name": "users_db", "user": {"@id": "101", ...}, "tag": ["active", "verified"]}}'

jsonify.dumps(xml, pretty=True, indent="  ")   # multi-line, 2-space indent

In-memory entry points

pygixml.jsonify.dumps(source, attr_prefix='@', cdata_key='#text', force_list=None, pretty=False, indent='\\t', encoding='utf-8')

Smart dispatcher: serializes XML to JSON regardless of what form the XML is already in:

Parameters:

  • source -- XML string, an already-parsed ObjectifiedElement, or an XMLNode.

  • attr_prefix (str) -- Prefix prepended to attribute keys. Default "@".

  • cdata_key (str) -- Key used for text content in mixed nodes. Default "#text".

  • force_list (set or True or None) -- Tag names that should always be wrapped in a JSON array, even when only one sibling exists. Pass True to force every tag.

  • pretty (bool) -- Indent the output. Default False (compact).

  • indent (str) -- Indentation string used when pretty=True. Default a tab.

Returns:

JSON string.

Return type:

str

Raises:

  • PygiXMLError -- If the XML is malformed.

  • TypeError -- If source’s type isn’t recognized.

  • ValueError -- If source is a str that doesn’t look like XML (file paths are rejected here on purpose: use dumps_file() explicitly for files).

Note

File input is intentionally excluded from the dispatcher: call dumps_file() directly for a path, so it’s always unambiguous whether a str argument is XML content or a file path.

pygixml.jsonify.dumps_str(xml, attr_prefix='@', cdata_key='#text', force_list=None, pretty=False, indent='\\t', encoding='utf-8')

Parse an XML string and serialize it directly to JSON.

Parameters:

xml (str) -- XML source text.

Returns:

JSON string.

Return type:

str

Raises:

PygiXMLError -- If the XML is malformed.

pygixml.jsonify.dumps_file(path, attr_prefix='@', cdata_key='#text', force_list=None, pretty=False, indent='\\t', encoding='utf-8')

Parse an XML file and serialize it directly to JSON, returning the result as a str. For files too big to hold as a JSON string in memory, see stream_dump() instead.

Parameters:

path (str) -- Filesystem path to the XML file.

Returns:

JSON string.

Return type:

str

Raises:

PygiXMLError -- If the file cannot be read or the XML is malformed.

pygixml.jsonify.dumps_obj(elem, attr_prefix='@', cdata_key='#text', force_list=None, pretty=False, indent='\\t', encoding='utf-8')

Serialize an already-parsed ObjectifiedElement subtree directly to JSON, without re-parsing or re-traversing as a dict first.

Parameters:

elem (pygixml.ObjectifiedElement) -- Element to serialize.

Returns:

JSON string.

Return type:

str

Raises:

TypeError -- If elem is not an ObjectifiedElement.

from pygixml import objectify, jsonify

root = objectify.from_string(xml)
jsonify.dumps_obj(root.user)            # just the <user> subtree
pygixml.jsonify.dumps_node(node, attr_prefix='@', cdata_key='#text', force_list=None, pretty=False, indent='\\t', encoding='utf-8')

Serialize a low-level XMLNode directly to JSON.

Parameters:

node (pygixml.XMLNode) -- Node to serialize.

Returns:

JSON string.

Return type:

str

Raises:

TypeError -- If node is not an XMLNode.

Streaming entry points: constant memory, files in and out

The functions above all hold the result (and, except for dumps_obj/dumps_node, the parsed DOM too) in memory: fine for documents that fit comfortably in RAM. For documents that don’t, jsonify has two streaming converters that go file-to-file, entirely in C++, with no pugixml DOM, no Python dict/list/str for individual elements, and no json module anywhere in the call chain.

pygixml.jsonify.stream_dump(xml_path, json_path, attr_prefix='@', cdata_key='#text', force_list=None, indent=0, stack_size=4096, io_buf_size=65536)

Convert a (potentially gigantic) XML file into a single, standard, valid JSON file: in roughly constant memory. Produces exactly what dumps_file() would produce (one JSON value mirroring the whole document, loadable with a plain json.load), just without ever holding the document, or the output, fully in memory.

Parameters:

  • xml_path (str) -- Path to the input XML file.

  • json_path (str) -- Path to the output JSON file. Overwritten if it exists.

  • attr_prefix (str) -- Prefix for XML attribute names in JSON keys. Default "@".

  • cdata_key (str) -- JSON key used for text content mixed with attributes or child elements. Default "#text".

  • force_list (set or True or None) -- Tag names always serialized as a JSON array. True forces every tag. Default None (a tag becomes an array only once a second sibling with that name actually appears).

  • indent (int) -- Spaces per nesting level, same convention as json.dump(..., indent=N). 0 (default) is compact; any positive value pretty-prints.

  • stack_size (int) -- Size in bytes of yxml’s internal name stack.

  • io_buf_size (int) -- Bytes read per XML I/O operation. Default 64 KB.

Returns:

Number of XML elements processed (informational).

Return type:

int

Raises:

PygiXMLError -- On malformed XML, or if the input/output file cannot be opened.

How it stays constant-memory while still producing valid JSON syntax. A JSON array needs to know, before its closing ], whether more items follow: but the parser only finds that out when (and if) a second same-tag sibling actually shows up. Rather than buffer whole subtrees to be safe, the engine writes optimistically and patches the output file in place once it learns more:

  • The first time a child tag is seen under some parent, nothing extra is written at all: just its value, as a plain (non-array) field. No placeholder, no reserved byte; a tag that never repeats costs nothing beyond its own content.

  • A second sibling with the same tag arrives:

    • If it’s adjacent to the first (the common case for record-oriented XML: same-tag siblings next to each other in the source), the engine inserts the opening [ and the new value right where the first value ended: an O(1) operation, and the only one this case ever needs.

    • If a different child tag was interleaved in between, the engine splices: it shifts the interleaved bytes forward (in fixed-size chunks) to open room for [ and the new value. Cost is proportional to how much was interleaved that one time, not to the file size.

Time complexity in practice. For the shape almost all record-oriented giant XML actually has: one repeated tag per nesting level, as in <orders><order>...</order>...</orders>: every sibling after the first lands via the O(1) adjacent case above, so the whole conversion is O(n). It degrades towards O(n²) only in an adversarial shape: two or more different tags repeating and interleaving at the same level for many iterations (e.g. <a>1</a><b>1</b><a>2</a><b>2</b>... with no wrapping element around each pair): there, every catch-up splice for one tag has to shift past the other tag’s entire, still-growing field, and that cost compounds across iterations. If that’s genuinely your data’s shape, stream_jsonl() below sidesteps the problem entirely by not trying to preserve a single JSON document at all. Memory stays O(1): a small fixed scratch buffer plus one bookkeeping entry per currently-open distinct tag: regardless of which case applies; only the time bound changes.

from pygixml import jsonify
import json

jsonify.stream_dump("huge.xml", "huge.json")            # compact
jsonify.stream_dump("huge.xml", "huge.json", indent=2)  # pretty

with open("huge.json") as f:
    data = json.load(f)   # a single, ordinary, valid JSON document
pygixml.jsonify.stream_jsonl(xml_path, jsonl_path, tag, attr_prefix='@', cdata_key='#text', force_list=None, stack_size=4096, io_buf_size=65536)

The file-to-file counterpart of iterjsonl() (and the per-tag, filtering complement to stream_dump(), which always converts the entire document): streams an XML file straight to a .jsonl file, one matched element per line, entirely in C++. Unlike iterjsonl(), no StreamElement and no Python str/dict/list is ever created for the matched elements themselves: each element’s JSON object is assembled in a small in-memory buffer (bounded by that one element’s own subtree, the same constant-memory model as iterfind()) and written straight to the file.

Parameters:

  • xml_path (str) -- Path to the source XML file.

  • jsonl_path (str) -- Path to the .jsonl file to write. Overwritten if it exists.

  • tag (str) -- Tag name of the elements to convert and write, one per line.

  • attr_prefix -- Same meaning as stream_dump().

  • cdata_key -- Same meaning as stream_dump().

  • force_list -- Same meaning as stream_dump().

  • stack_size (int) -- Size in bytes of yxml’s internal name stack.

  • io_buf_size (int) -- Bytes read per XML I/O operation. Default 64 KB.

Returns:

Number of matched elements written.

Return type:

int

Raises:

PygiXMLError -- On malformed XML, or if the input/output file cannot be opened.

Note

If tag appears nested inside an already-matched element, that inner occurrence is folded into the outer match as an ordinary nested field (under its own tag-name key) rather than written as a second, separate line: only the outermost occurrence of a match starts a new JSONL record. This only matters for genuinely self-nested tags; a flat list of repeated sibling records (the common case) is unaffected.

from pygixml import jsonify

n = jsonify.stream_jsonl("huge.xml", "huge.jsonl", "record")
print(f"wrote {n} records")

import json
with open("huge.jsonl") as f:
    for line in f:
        record = json.loads(line)   # each line is independent, valid JSON

Choosing the right entry point

You have / want

In memory

Streamed (constant memory)

Whole document → one JSON value

dumps() / dumps_file()

stream_dump()

One record per line, output as a .jsonl file

Loop + write yourself (see iterjsonl())

stream_jsonl()

One record per line, kept as Python str objects

iterjsonl()

(inherently produces a Python object per line: see Streaming: Constant-Memory Parsing for Big XML for the all-C++ alternative when that’s not needed)

Already-parsed ObjectifiedElement / XMLNode

dumps_obj() / dumps_node()

(n/a: already in memory)

See Streaming: Constant-Memory Parsing for Big XML for iterjsonl and the rest of the underlying constant-memory parsing layer that stream_dump and stream_jsonl are built on top of.