stefan
/
tpcpr
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

updated TODOs

This commit is contained in:
gurkenhabicht 2023-08-03 22:50:27 +02:00
parent 1e6fc92e41
commit 8367067d3d
1 changed files with 5 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
README.md
View File

@ -3,7 +3,7 @@
The software is written in Rust (2018, safe mode only). At the current state I am having fun writing in Rust and testing language features. The code should be modular enough to change any function you deem awful enough. The software is written in Rust (2018, safe mode only). At the current state I am having fun writing in Rust and testing language features. The code should be modular enough to change any function you deem awful enough.
Error handling is subpar at the moment. There is no real unit testing to speak of since switching to asynchronous functionality. Testing will come back. Error handling is subpar at the moment. There is no real unit testing to speak of since switching to asynchronous functionality. Testing will come back.
This version is a successor of the _POSIX_C_SOURCE 200809L implementation in which all data parsed from a pcap/pcapng files is written as a single and simple query. The ingestion time is rather fast (tested writes: 100*10^3 tcp packets in ~1.8 sec) but the procedure may be insecure. See the other repository for more information. This version is a successor of the _POSIX_C_SOURCE 200809L_ implementation in which all data parsed from a pcap/pcapng files is written as a single and simple query. The ingestion time is rather fast (tested writes: 100*10^3 tcp packets in ~1.8 sec) but the procedure may be insecure. See the other repository for more information.
~~The idea of this iteration is to use a prepared statement and chunk the data according to maximum input. Postgres databases have a custom maximum limit on each insert query of prepared statements. Said chunk size is initialized through the config/interface file called parser.json as `insert_max`. Data can be read from PCAP/PCANG files, as well as network devices.~~ ~~The idea of this iteration is to use a prepared statement and chunk the data according to maximum input. Postgres databases have a custom maximum limit on each insert query of prepared statements. Said chunk size is initialized through the config/interface file called parser.json as `insert_max`. Data can be read from PCAP/PCANG files, as well as network devices.~~
**UPDATE 0.2.0**: Chunking can be omitted completely when using PostgreSQL's `COPY` transferring binary data instead of using `Insert`. This is not only somewhat faster, but there are quite a few lines of code less in the end. Only parsing from network device uses needs chunks, at the moment. **UPDATE 0.2.0**: Chunking can be omitted completely when using PostgreSQL's `COPY` transferring binary data instead of using `Insert`. This is not only somewhat faster, but there are quite a few lines of code less in the end. Only parsing from network device uses needs chunks, at the moment.
@ -19,10 +19,13 @@ Caveats: Regex Syntax is limited and needs soome love. Escaping common regular
If this whole thing turns out to be viable, some future features may be: If this whole thing turns out to be viable, some future features may be:
- InfluxDB and SQLite output
- Protobuf serialization
- More network protocols
- Database containing the already implemented file hash map to compare file status/sizes after the parser may have crashed, or to join a complete overview of any existing PCAP files inserted at previous CTFs. - Database containing the already implemented file hash map to compare file status/sizes after the parser may have crashed, or to join a complete overview of any existing PCAP files inserted at previous CTFs.
- Update file hashmap through inotify crate, during runtime. - Update file hashmap through inotify crate, during runtime.
- Restoration of fragmented ipv4 packages. - Restoration of fragmented ipv4 packages.
- SIMD (via autovectorization). Which is easy enough to do in Rust. - SIMD (via autovectorization)
- Support more network protocols - Support more network protocols
There are many other things left to be desired. There are many other things left to be desired.