ad37c0b97c
Hi rust enthusiasts,
With this patch I propose to add a "streaming" API to the existing json parser in libserialize.
By "streaming" I mean a parser that let you act on JsonEvents that are generated as while parsing happens, as opposed to parsing the entire source, generating a big data structure and working with this data structure. I think both approaches have their pros and cons so this pull request adds the streaming API, preserving the existing one.
The streaming API is simple: It consist into an Iterator<JsonEvent> that consumes an Iterator<char>. JsonEvent is an enum with values such as NumberValue(f64), BeginList, EndList, BeginObject, etc.
The user would ideally use the API as follows:
```
for evt in StreamingParser::new(src) {
match evt {
BeginList => {
// ...
}
// ...
}
}
```
The iterator provides a stack() method returning a slice of StackNodes which represent "where we currently are" in the logical structure of the json stream (for instance at "foo.bar[3].x" you get [ Key("foo"), Key("bar"), Index(3), Key("x") ].)
I wrote "ideally" above because the current way rust expands for loops, you can't call the stack() method because the iterator is already borrowed. So for know you need to manually advance the iterator in the loop. I hope this is something we can cope with, until for loops are better integrated with the compiler.
Streaming parsers are useful when you want to read from a json stream, generate a custom data structure and you know how the json is going to be structured. For example, imagine you have to parse a 3D mesh file represented in the json format. In this case you probably expect to have large arrays of vertices and using the generic parser will be very inefficient because it will create a big list of all these vertices, which you will copy into a contiguous array afterwards (so you end up doing a lot of small allocations, parsing the json once and parsing the data structure afterwards). With a streaming parser, you can add the vertices to a contiguous array as they come in without paying the cost of creating the intermediate Json data structure. You have much fewer allocations since you write directly in the final data structure and you can be smart in how you will pre-allocate it.
I added added this directly into serialize::json rather than in its own library because it turns out I can reuse most of the existing code whereas maintaining a separate library (which I did originally) forces me to duplicate this code.
I wrote this trying to minimize the size of the patch so there may be places where the code could be nicer at the expenses of more changes (let me know what you prefer).
This is my first (potential) contribution to rust, so please let me know if I am doing something wrong (maybe I should have first introduced this proposition in the mailing list, or opened a github issue, etc.?). I work a few meters away from @pknfelix so I am not too hard to find :)
|
||
|---|---|---|
| man | ||
| mk | ||
| src | ||
| .gitattributes | ||
| .gitignore | ||
| .gitmodules | ||
| .mailmap | ||
| .travis.yml | ||
| AUTHORS.txt | ||
| configure | ||
| CONTRIBUTING.md | ||
| COPYRIGHT | ||
| LICENSE-APACHE | ||
| LICENSE-MIT | ||
| Makefile.in | ||
| README.md | ||
| RELEASES.txt | ||
The Rust Programming Language
This is a compiler for Rust, including standard libraries, tools and documentation.
Quick Start
- Download a binary installer for your platform.
- Read the tutorial.
- Enjoy!
Note: Windows users can read the detailed getting started notes on the wiki.
Building from Source
-
Make sure you have installed the dependencies:
g++4.4 orclang++3.xpython2.6 or later (but not 3.x)perl5.0 or later- GNU
make3.81 or later curlgit
-
Download and build Rust:
You can either download a tarball or build directly from the repo.
To build from the tarball do:
$ curl -O http://static.rust-lang.org/dist/rust-nightly.tar.gz $ tar -xzf rust-nightly.tar.gz $ cd rust-nightlyOr to build from the repo do:
$ git clone https://github.com/mozilla/rust.git $ cd rustNow that you have Rust's source code, you can configure and build it:
$ ./configure $ make && make installNote: You may need to use
sudo make installif you do not normally have permission to modify the destination directory. The install locations can be adjusted by passing a--prefixargument toconfigure. Various other options are also supported, pass--helpfor more information on them.When complete,
make installwill place several programs into/usr/local/bin:rustc, the Rust compiler, andrustdoc, the API-documentation tool. system. -
Read the tutorial.
-
Enjoy!
Notes
Since the Rust compiler is written in Rust, it must be built by a precompiled "snapshot" version of itself (made in an earlier state of development). As such, source builds require a connection to the Internet, to fetch snapshots, and an OS that can execute the available snapshot binaries.
Snapshot binaries are currently built and tested on several platforms:
- Windows (7, 8, Server 2008 R2), x86 only
- Linux (2.6.18 or later, various distributions), x86 and x86-64
- OSX 10.7 (Lion) or greater, x86 and x86-64
You may find that other platforms work, but these are our officially supported build environments that are most likely to work.
Rust currently needs about 1.5 GiB of RAM to build without swapping; if it hits swap, it will take a very long time to build.
There is a lot more documentation in the wiki.
License
Rust is primarily distributed under the terms of both the MIT license and the Apache License (Version 2.0), with portions covered by various BSD-like licenses.
See LICENSE-APACHE, LICENSE-MIT, and COPYRIGHT for details.