pool/pmacct
SHA256
4
0
Fork 1
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
pmacct/pmacct.changes

747 lines
42 KiB
Plaintext
Raw Blame History

-------------------------------------------------------------------
Sun Oct 22 09:22:28 UTC 2017 - mardnh@gmx.de
- update to version 1.7.0
+ ZeroMQ integration: by defining plugin_pipe_zmq to 'true', ZeroMQ is
used for queueing between the Core Process and plugins. This is in
alternative to the home-grown circular queue implementation (ie.
plugin_pipe_size). plugin_pipe_zmq_profile can be set to one value
of { micro, small, medium, large, xlarge } and allows to select
among a few standard buffering profiles without having to fiddle
with plugin_buffer_size. How to compile, install and operate ZeroMQ
is documented in the "Internal buffering and queueing" section of
the QUICKSTART document.
+ nDPI integration: enables packet classification, replacing existing
L7-layer project integration, and is available for pmacctd and
uacctd. The feature, once nDPI is compiled in, is simply enabled by
specifying 'class' as part of the aggregation method. How to compile
install and operate nDPI is documented in the "Quickstart guide to
packet classification" section of the QUICKSTART document.
+ nfacctd: introduced nfacctd_templates_file so that NetFlow v9/IPFIX
templates can be cached to disk to limit the amount of lost packets
due to unknown templates when nfacctd (re)starts. The implementation
is courtesy by Codethink Ltd.
+ nfacctd: introduced support for PEN on IPFIX option templates. This
is in addition to already supported PEN for data templates. Thanks
to Gilad Zamoshinski ( @zamog ) for his support.
+ sfacctd: introduced new aggregation primitives (tunnel_src_host,
tunnel_dst_host, tunnel_proto, tunnel_tos) to support inner L3
layers. Thanks to Kaname Nishizuka ( @__kaname__ ) for his support.
+ nfacctd, sfacctd: pcap_savefile and pcap_savefile_wait were ported
from pmacctd. They allow to process NetFlow/IPFIX and sFlow data
from previously captured packets; these also ease some debugging by
not having to resort anymore to tcpreplay for most cases.
+ pmacctd, sfacctd: nfacctd_time_new feature has been ported so, when
historical accounting is enabled, to allow to choose among capture
time and time of receipt at the collector for time-binning.
+ nfacctd: added support for NetFlow v9/IPFIX field types #130/#131,
respectively the IPv4/IPv6 address of the element exporter.
+ nfacctd: introduced nfacctd_disable_opt_scope_check: mainly a work
around to implementations not encoding NetFlow v9/IPIFX option scope
correctly, this knob allows to disable option scope checking. Thanks
to Gilad Zamoshinski ( @zamog ) for his support.
+ pre_tag_map: added 'source_id' key for tagging on NetFlow v9/IPFIX
source_id field. Added also 'fwdstatus' for tagging on NetFlow v9/
IPFIX information element #89: this implementation is courtesy by
Emil Palm ( @mrevilme ).
+ tee plugin: tagging is now possible on NetFlow v5-v8 engine_type/
engine_id, NetFlow v9/IPFIX source_id and sFlow AgentId.
+ tee plugin: added support for 'src_port' in tee_receivers map. When
in non-transparent replication mode, use the specified UDP port to
send data to receiver(s). This is in addition to tee_source_ip,
which allows to set a configured IP address as source.
+ networks_no_mask_if_zero: a new knob so that IP prefixes with zero
mask - that is, unknown ones or those hitting a default route - are
not masked. The feature applies to *_net aggregation primitives and
makes sure individual IP addresses belonging to unknown IP prefixes
are not zeroed out.
+ networks_file: hooked up networks_file_no_lpm feature to peer and
origin ASNs and (BGP) next-hop fields.
+ pmacctd: added support for calling pcap_set_protocol() if supported
by libpcap. Patch is courtesy by Lennert Buytenhek ( @buytenh ).
+ pmbgpd, pmbmpd, pmtelemetryd: added a few CL options to ease output
of BGP, BMP and Streaming Telemetry data, for example: -o supplies
a b[gm]p_daemon_msglog_file, -O supplies a b[gm]p_dump_file and -i
supplies b[gm]p_dump_refresh_time.
+ kafka plugin: in the examples section, added a Kafka consumer script
using the performing confluent-kafka-python module.
! fix, BGP daemon: segfault with add-path enabled peers as per issue
#128. Patch is courtesy by Markus Weber ( @FvDxxx ).
! fix, print plugin: do not update link to latest file if cause of
purging is a safe action (ie. cache space is finished. Thanks to
Camilo Cardona ( @jccardonar ) for reporting the issue. Also, for
the same reason, do not execute triggers (ie. print_trigger_exec).
! fix, nfacctd: improved IP protocol check in NF_evaluate_flow_type()
A missing length check was causing, under certain conditions, some
flows to be marked as IPv6. Many thanks to Yann Belin for his
support resolving the issue.
! fix, print and SQL plugins: optimized the cases when the dynamic
filename/table has to be re-evaluated. This results in purge speed
gains when the dynamic part is time-related and nfacctd_time_new is
set to true.
! fix, bgp_daemon_md5_file: if the server socket is AF_INET and the
compared peer address in MD5 file is AF_INET6 (v4-mapped v6), pass
it through ipv4_mapped_to_ipv4(). Also if the server socket is
AF_INET6 and the compared peer addess in MD5 file is AF_INET, pass
it through ipv4_to_ipv4_mapped(). Thanks to Paul Mabey for reporting
the issue.
! fix, nfacctd: improved length checks in resolve_vlen_template() to
prevent SEGVs. Thanks to Josh Suhr and Levi Mason for their support.
! fix, nfacctd: flow stitching, improved flow end time checks. Thanks
to Fabio Bindi ( @FabioLiv ) for his support resolving the issue.
! fix, amqp_common.c: amqp_persistent_msg now declares the RabbitMQ
exchange as durable in addition to marking messages as persistent;
this is related to issue #148.
! fix, nfacctd: added flowset count check to existing length checks
for NetFlow v9/IPFIX datagrams. This is to avoid logs flooding in
case of padding. Thanks to Steffen Plotner for reporting the issue.
! fix, BGP daemon: when dumping BGP data at regular time intervals,
dump_close message contained wrongly formatted timestamp. Thanks to
Yuri Lachin for reporting the issue.
! fix, MySQL plugin: if --enable-ipv6 and sql_num_hosts set to true,
use INET6_ATON for both v4 and v6 addresses. Thanks to Guy Lowe
( @gunkaaa ) for reporting the issue and his support resolving it.
! fix, 'flows' primitive: it has been wired to sFlow so to count Flow
Samples received. This is to support Q21 in FAQS document.
! fix, BGP daemon: Extended Communities value was printed with %d
(signed) format string instead of %u (unsigned), causing issue on
large values.
! fix, aggregate_primitives: improved support of 'u_int' semantics for
8 bytes integers. This is in addition to already supported 1, 2 and
4 bytes integers.
! fix, pidfile: pidfile created by plugin processes was not removed.
Thanks to Yuri Lachin for reporting the issue.
! fix, print plugin: checking non-null file descriptor before setvbuf
in order to prevent SEGV. Similar checks were added to prevent nulls
be input to libavro calls when Apache Avro output is selected.
! fix, SQL plugins: MPLS aggregation primitives were not correctly
activated in case sql_optimize_clauses was set to false.
! fix, building system: reviewed minimum requirement for libraries,
removed unused m4 macros, split features in plugins (ie. MySQL) and
supports (ie. JSON).
! fix, sql_history: it now correctly honors periods expressed is 's'
seconds.
! fix, BGP daemon: rewritten bgp_peer_print() to be thread safe.
! fix, pretag.h: addressed compiler warning on 32-bit architectures,
integer constant is too large for "long" type. Thanks to Stephen
Clark ( @sclark46 ) for reporting the issue.
- MongoDB plugin: it is being discontinued since the old Mongo API is
not supported anymore and there has never been enough push from the
community to transition to the new/current API (which would require
a rewrite of most of the plugin). In this phase-1 the existing
MongoDB plugin is still available using 'plugins: mongodb_legacy'
in the configuration.
- Packet classification basing on the L7-filter project is being
discontinued (ie. 'classifiers' directive). This is being replaced
by an implementation basing on the nDPI project. As part of this
also the sql_aggressive_classification knob has been discontinued.
- tee_receiver was part of the original implementation of the tee
plugin, allowing to forward to a single target and hence requiring
multiple plugins instantiated, one per target. Since 0.14.3 this
directive was effectively outdated by tee_receivers.
- tmp_net_own_field: the knob has been discontinued and was allowing
to revert to backward compatible behaviour of IP prefixes (ie.
src_net) being written in the same field as IP addresses (ie.
src_host).
- tmp_comms_same_field: the knob has been discontinued and was
allowing to revert to backward compatible behaviour of BGP
communities (standard and extended) being writeen all in the same
field.
- plugin_pipe_amqp and plugin_pipe_kafka features were meant as an
alternative to the homegrown queue solution for internal messaging,
ie. passing data from the Core Process to Plugins, and are being
discontinued. They are being replaced by a new implementation,
plugin_pipe_zmq, basing on ZeroMQ.
- plugin_pipe_backlog was allowing to keep an artificial backlog of
data in the Core Process so to maximise bypass poll() syscalls in
plugins. If home-grown queueing is found limiting, instead of
falling back to such strategies, ZeroMQ queueing should be used.
- pmacctd: deprecated support for legacy link layers: FDDI, Token Ring
and HDLC.
-------------------------------------------------------------------
Sat Apr 22 18:12:00 UTC 2017 - mardnh@gmx.de
- update to version 1.6.2
+ BGP, BMP daemons: introduced support for BGP Large Communities IETF
draft (draft-ietf-idr-large-community). Large Communities are stored
in a variable-length field. Thanks to Job Snijders ( @job ) for his
support.
+ BGP daemon: implemented draft-ietf-idr-shutdown. The draft defines a
mechanism to transmit a short freeform UTF-8 message as part of a
Cease NOTIFICATION message to inform the peer why the BGP session is
being shutdown or reset. Thanks to Job Snijders ( @job ) for his
support.
+ tee plugin, pre_tag_map: introduced support for inspetion of specific
flow primitives and selective replication over them. The primitives
supported are: input and output interfaces, source and destination
MAC addresses, VLAN ID. The feature is now limited to sFlow v5 only.
Thanks to Nick Hilliard and Barry O'Donovan for their support.
+ Added src_host_pocode and dst_host_pocode primitives, pocode being a
compact and (de-)aggregatable (easy to identify districts, cities,
metro areas, etc.) geographical representation, based on the Maxmind
v2 City Database. Thanks to Jerred Horsman for his support.
+ Kafka support: introduced support for user-defined (librdkafka) config
file via the new *_kafka_config_file config directives. Full pathname
to a file containing directives to configure librdkafka is expected.
All knobs whose values are string, integer, boolean are supported.
+ AMQP, Kafka plugins: introduced new directives kafka_avro_schema_topic,
amqp_avro_schema_routing_key to transmit Apache Avro schemas at regular
time intervals. The routing key/topic can overlap with the one used to
send actual data.
+ AMQP, Kafka plugins: introduced support for start/stop markers when
encoding is set to Avro (ie. 'kafka_output: avro'); also Avro schema
is now embedded in a JSON envelope when sending it via a topic/routing
key (ie. kafka_avro_schema_topic).
+ print plugin: introduced new config directive avro_schema_output_file
to save the Apache Avro schema in a separate file (it was only possible
to have it combined at the beginning of the data file).
+ BGP daemon: introduced a new bgp_daemon_as config directive to set a
LocalAS which could be different from the remote peer one. This is to
establish an eBGP session instead of a iBGP one (default).
+ flow_to_rd_map: introduced support for mpls_vpn_id. In NetFlow/IPFIX
this is compared against Field Types #234 and #235.
+ sfacctd: introduced support for sFlow v2/v4 counter samples (generic,
ethernet, vlan). This is in addition to existing support for sFlow v5
counters.
+ BGP, BMP and Streming Telemetry daemons: added writer_id field when
writing to Kafka and/or RabbitMQ. The field reports the configured
core_proc_name and the actual PID of the writer process (so, while
being able to correlate writes to the same daemon, it's also possible
to distinguish among overlapping writes).
+ amqp, kafka, print plugins: harmonized JSON output to the above: added
event_type field, writer_id field with plugin name and PID.
+ BGP, BMP daemons: added AFI, SAFI information to log and dump outputs;
also show VPN Label if SAFI is MPLS VPN.
+ pmbgpd, pmbmpd: added logics to bypass building RIBs if only logging
BGP/BMP data real-time.
+ BMP daemon: added BMP peer TCP port to log and dump outputs (for NAT
traversal scenarios). Contextually, multiple TCP sessions per IP are
now supported for the same reason.
+ SQL plugins: ported (from print, etc. plugins) the 1.6.1 re-working of
the max_writers feature.
+ uacctd: use current time when we don't have a timestamp from netlink.
We only get a timestamp when there is a timestamp in the skb. Notably,
locally generated packets don't get a timestamp. The patch is courtesy
by Vincent Bernat ( @vincentbernat ).
+ build system: added configure options for partial linking of binaries
with any selection/combination of IPv4/IPv6 accounting daemons, BGP
daemon, BMP daemon and Streaming Telemetry daemon possible. By default
all are compiled in.
+ BMP daemon: internal code changes to pass additional info from BMP
per-peer header to bgp_parse_update_msg(). Goal is to expose further
info, ie. pre- vs post- policy, when logging or dumping BMP info.
! fix, BGP daemon: introduced parsing of IPv6 MPLS VPN (vpnv6) NLRIs.
Thanks to Alberto Santos ( @m4ccbr ) for reporting the issue.
! fix, BGP daemon: upon doing routes lookup, now correctly honouring
the case of BGP-LU (SAFI_MPLS_LABEL).
! fix, BGP daemon: send BGP NOTIFICATION out in case of known failures
in bgp_parse_msg().
! fix, kafka_partition, *_kafka_partition: default value changed from 0
(partition zero) to -1 (RD_KAFKA_PARTITION_UA, partition unassigned).
Thanks to Johan van den Dorpe ( @johanek ) for his support.
! fix, pre_tag_map: removed constraint for 'ip' keyword for nfacctd and
sfacctd maps. While this is equivalent syntax to specifying rules with
'ip=0.0.0.0/0', it allows for map indexing (maps_index: true).
! fix, bgp_agent_map: improved sanity check against bgp_ip for IPv6
addresses (ie. an issue appeared for the case of '::1' where the first
64 bits are zeroed out). Thanks to Charlie Smurthwaite ( @catphish )
for reporting the issue.
! fix, maps_index: indexing now correctly works for IPv6 pre_tag_map
entries. That is, those where 'ip', the IP address of the NetFlow/
IPFIX/sFlow exporter, is an IPv6 address.
! fix, pre_tag_map: if mpls_vpn_rd matching condition is specified and
maps_index is enabled, PT_map_index_fdata_mpls_vpn_rd_handler() now
picks the right (and expected) info.
! fix, pkt_handlers.c: improved definition and condition to free() in
bgp_ext_handler() in order to prevent SEGVs. Thanks to Paul Mabey for
his support.
! fix, kafka_common.c: removed waiting time from p_kafka_set_topic().
Added docs advicing to create in advance Kafka topics.
! fix, sfacctd, sfprobe: tag and tag2 are now correctly re-defined as
64 bits long.
! fix, sfprobe plugin, sfacctd: tags and class primitives are now being
encoded/decoded using enterprise #43874, legit, instead of #8800, that
was squatted back in the times. See issue #71 on GiHub for more info.
! fix, sfacctd: lengthCheck() + skipBytes() were producing an incorrect
jump in case of unknown flow samples. Replaced by skipBytesAndCheck().
Thanks to Elisa Jasinska ( @fooelisa ) for her support.
! fix, pretag_handlers.c: in bgp_agent_map added case for 'vlan and ...'
filter values.
! fix, BGP daemon: multiple issues of partial visibility of the stored
RIBs and SEGVs when bgp_table_per_peer_buckets was not left default:
don't mess with bms->table_per_peer_buckets given the multi-threaded
scenario. Thanks to Dan Berger ( @dfberger ) for his support.
! fix, BGP, BMP daemons: bgp_process_withdraw() function init aligned to
bgp_process_update() in order to prevent SEGVs. Thanks to Yuri Lachin
for his support.
! fix, bgp_msg.c: Route Distinguisher was stored and printed incorrectly
when of type RD_TYPE_IP. Thanks to Alberto Santos ( @m4ccbr ) for
reporting the issue.
! fix, bgp_logdump.c: p_kafka_set_topic() was being wrongly applied to
an amqp_host structure (instead of a kafka_host structure). Thanks to
Corentin Neau ( @weyfonk ) for reporting the issue.
! fix, BGP daemon: improved BGP next-hop setting and comparison in cases
of MP_REACH_NLRI and MPLS VPNs. Many thanks to both Catalin Petrescu
( @cpmarvin ) and Alberto Santos ( @m4ccbr ) for their support.
! fix, pmbgpd, pmbmpd: pidfile was not written even if configured. Thanks
to Aaron Glenn ( @aaglenn ) for reporting the issue.
! fix, tee plugin: tee_max_receiver_pools is now correctly honoured and
debug message shows the replicatd protocol, ie. NetFlow/IPFIX vs sFlow.
! AMQP, Kafka plugins: separate JSON objects, newline separated, are
preferred to JSON arrays when buffering of output is enabled (ie.
kafka_multi_values) and output is set to JSON. This is due to quicker
serialisation performance shown by the Jansson library.
! build system: switched to enable IPv6 support by default (while the
--disable-ipv6 knob can be used to reverse the behaviour). Patch is
courtesy by Elisa Jasinska ( @fooelisa ).
! build system: given visibility, ie. via -V CL option, into compile
options enabled by default (ie. IPv6, threads, 64bit counters, etc.).
! fix, nfprobe: free expired records when exporting to an unavailable
collector in order to prevent a memory leak. Patch is courtersy by
Vladimir Kunschikov ( @kunschikov ).
! fix, AMQP plugin: set content type to binary in case of Apache Avro
output.
! fix, AMQP, Kafka plugins: optimized amqp_avro_schema_routing_key and
kafka_avro_schema_topic. Avro schema is built only once at startup.
! fix, cfg.c: improved parsing of config key-values where squared brakets
appear in the value part. Thanks to Brad Hein ( @regulatre ) for
reporting the issue. Also, detection of duplicates among plugin and
core process names was improved.
! fix, misc: compiler warnings: fix up missing includes and prototypes;
the patch is courtesy by Tim LaBerge ( @tlaberge ).
! kafka_consumer.py, amqp_receiver.py: Kafka, RabbitMQ consumer example
scripts have been greatly expanded to support posting to a REST API or
to a new Kafka topic, including some stats. Also conversion of multiple
newline-separated JSON objects to a JSON array has been added. Misc
bugs were fixed.
- remove patcch: pmacct-fix-implicit-pointer-decl.diff
-------------------------------------------------------------------
Wed Jul 13 10:22:52 UTC 2016 - mardnh@gmx.de
- add systemd scripts
- add manpage for pmacct
- remove not longer supported build options
- enable-v4-mapped
- with-pgsql-includes
- fix build for older SUSE versions (SLES11SP4, SLES12, OpenSUSE 13.1)
- add patch for psql-header detection on SLES11SP4 and openSUSE 13.1
- pmacct-pgsql-fix-header-detection-without-autoreconf.diff
-------------------------------------------------------------------
Sat Jun 11 17:50:42 UTC 2016 - mardnh@gmx.de
- update to version 1.6.0
+ Streamed telemetry daemon: quoting Cisco IOS-XR Telemetry Configuration
Guide at the time of this writing: "Streaming telemetry [ .. ] data
can be used for analysis and troubleshooting purposes to maintain the
health of the network. This is achieved by leveraging the capabilities of
machine-to-machine communication. [ .. ]" Streamed telemetry support comes
in two flavours: 1) a telemetry thread can be started in existing daemons,
ie. sFlow, NetFlow/IPFIX, etc. for the purpose of data correlation and 2)
a new daemon pmtelemetryd for standalone consumpton of data. Streamed
telemetry data can be logged real-time and/or dumped at regular time
intervals to flat-files, RabbitMQ or Kafka brokers.
+ BMP daemon: introduced support for Route Monitoring messages. RM messages
"provide an initial dump of all routes received from a peer as well as an
ongoing mechanism that sends the incremental routes advertised and
withdrawn by a peer to the monitoring station". Like for BMP events, RM
messages can be logged real-time and/or dumped at regular time intervals
to flat-files, RabbiMQ and Kafka brokers. RM messages are also saved in a
RIB structure for IP prefix lookup.
+ uacctd: ULOG support switched to NFLOG, the newer and L3 independent Linux
packet logging framework. One of the key advantages of NFLOG is support for
IPv4 and IPv6 (whereas ULOG was restricted to IPv4 only). The code has been
contributed by Vincent Bernat ( @vincentbernat ).
+ build system: it was modernized so not to rely on specific and old versions
of automake and autoconf, as it was the case until 1.5. Among the things,
pkg-config and libtool are leveraged and an autogen.sh script is generated.
The code has been contributed by Vincent Bernat ( @vincentbernat ).
+ sfacctd: RabbitMQ and Kafka support was introduced to real-time log and/
or dump at regular time intervals of sFlow counters. This is in addition
to existing support for flat-files.
+ maps_index: several improvements were carried out in the area of indexing
of maps: optimizations to pretag_index_fill() and pretag_index_lookup() to
improve lookup speeds; optimized id_entry structure, ie. by splitting key
and non-key parts, and hashing key in order to consume less memory; added
duplicate entry detection (cause of sudden index destruction);
pretag_index_destroy() destroys hash keys for each index entry, solving a
memory leak issue. Thanks to Job Snijders ( @job ) for his support.
+ Introduced 'export_proto_seqno' aggregation primitive to report on
sequence number of the export protocol (ie. NetFlow, sFlow, IPFIX). This
feature may enable more advanced offline analysis of packet loss, out of
orders, etc. over time windows than basic online analytics provided by the
daemons.
+ log.c: logging moved from standard output (stdout) to standard error
(stderr) so to not conflict with stdout printing of statistics (print
plugin). Thanks to Jim Westfall ( @jwestfall69 ) for his support.
+ print plugin: introduced a new print_output_lock_file config directive
to lock standard output (stdout) output so to prevent multiple processes
(instances of the same print plugin or different instances of print plugin)
overlap output. Thanks to Jim Westfall ( @jwestfall69 ) for his support.
+ pkt_handlers.c: euristics in NetFlow v9/IPFIX VLAN handler were improved
for the case of flows in egress direction. Also IP protocol checks were
removed for UDP/TCP ports and TCP flags in case the export protocol is
NetFlow v9/IPFIX. Thanks to Alexander Ponamarchuk for his support.
! Code refactoring: improved re-usability of much of the BGP code (so to
make it possible to use it as a library for some BMP daemon features, ie.
Route Monitoring messages support); consolidated functions to handle log
and print plugin output files; improved log messages to always include
process name and type.
! fix, bpf_filter.c: issue compiling against libpcap 1.7.x; introduced a
check for existing bpf_filter() in libpcap in order to prevent namespace
conflicts.
! fix, tmp_net_own_field default value changed to true. This knob can be
still switched to false for this release but is going to be removed soon.
! fix, cfg.c, cfg_handlers.c, pmacct.c: some configuration directives and
pmacct CL parameters requiring string parsing, ie. -T -O -c, are now
passed through tolower().
! fix, MongoDB plugin: removed version check around mongo_create_index()
and now defaulting to latest MongoDB C legacy driver API. This is due to
some versioning issue in the driver.
! fix, timestamp_arrival: primitive was reporting incorrect results (ie.
always zero) if timestamp_start or timestamp_end were not also specified
as part of the same aggregation method. Many thanks to Vincent Morel for
reporting the issue.
! fix, thread stack: a value of 0, default, leaves the stack size to the
system default or pmacct minimum (8192000) if system default is too low.
Some systems may throw an error if the defined size is not a multiple of
the system page size.
! fix, nfacctd: improved NetFlow v9/IPFIX parsing. Added new length checks
and fixed some existing checks. Thanks to Robert Wuttke ( @Benocs ) for his
support.
! fix, pretag_handlers.c: BPAS_map_bgp_nexthop_handler() and BPAS_map_bgp_
peer_dst_as_handler() were not setting a func_type.
! fix, JSON support: Jansson 2.2 does not have json_object_update_missing()
function which was introduced in 2.3. This is not provided as part of a
jansson.c file and compiled in conditionally, if needed. Jansson 2.2 is
still shipped along by some recent OS releases. Thanks to Vincent Bernat
( @vincentbernat ) for contributing the patch.
! fix, log.c: use a format string when calling syslog(). Passing directly a
potentially uncontrolled string could crash the program if the string
contains formatting parameters. Thanks to Vincent Bernat ( @vincentbernat )
for contributing the patch.
! fix, sfacctd.c: default value for config.sfacctd_counter_max_nodes was set
after sf_cnt_link_misc_structs(). Thanks to Robin Douine for his support
resolving the issue.
! fix, sfacctd.c: timestamp was consistently being reported as null in sFlow
counters output. Thanks to Robin Douine for his support resolving the issue.
! fix, SQL plugins: $SQL_HISTORY_BASETIME environment variable was reporting a
wrong value (next basetime) in the sql_trigger_exec script. Thanks to Rain
Nõmm for reporting the issue.
! fix, pretag.c: in pretag_index_fill(), replaced memcpy() with hash_dup_key()
also a missing res_fdata initialization in pretag_index_lookup() was solved;
these issues were originating false negatives upon lookup. Thanks to Rain
Nõmm fo his suppor.
! fix, ISIS daemon: hash_* functions renamed into isis_hash_* to avoid name
space clashes with their BGP daemon counter-parts.
! fix, kafka_common.c: rd_kafka_conf_set_log_cb moved to p_kafka_init_host()
due to crashes seen in p_kafka_connect_to_produce(). Thanks to Paul Mabey
for his support resolving the issue.
! fix, bgp_lookup.c: bgp_node_match_* were not returning any match in
bgp_follow_nexthop_lookup(). Thanks to Tim Jackson ( @jackson-tim ) for his
support resolving the issue.
! fix, sql_common.c: crashes observed when nfacctd_stitching was set to true
and nfacctd_time_new was set to false. Thanks to Jaroslav Jiráse
( @jjirasek ) for his support solving the issue.
- SQL plugins: sql_recovery_logfile feature was removed from the code due
to lack of support and interest. Along with it, also pmmyplay and pmpgplay
tools have been removed.
- pre_tag_map: removed support for mpls_pw_id due to lack of interest.
-------------------------------------------------------------------
Thu Jan 14 18:59:13 UTC 2016 - mardnh@gmx.de
- update to version 1.5.3
+ Introduced the Kafka plugin: Apache Kafka is publish-subscribe messaging
rethought as a distributed commit log. Its qualities being: fast, scalable,
durable and distributed by design. pmacct Kafka plugin is designed to
send aggregated network traffic data, in JSON format, through a Kafka
broker to 3rd party applications.
+ Introduced Kafka support to BGP and BMP daemons, in both their msglog
and dump flavors (ie. see [bgp|bmp]_daemon_msglog_kafka_broker_host and
[bgp_table|bmp]_dump_kafka_broker_host and companion config directives).
+ Introduced support for a Kafka broker to be used for queueing and data
exchange between Core Process and plugins. plugin_pipe_kafka directive,
along with all other plugin_pipe_kafka_* directives, can be set globally
or apply on a per plugin basis - similarly to what was done for RabbitMQ
(ie. plugin_pipe_amqp). Support is currently restricted only to print
plugin.
+ Added a new timestamp_arrival primitive to expose NetFlow/IPFIX records
observation time (ie. arrival at the collector), in addition to flows
start and end times (timestamp_start and timestamp_end respectively).
+ plugin_pipe_amqp: feature extended to the plugins missing it: nfprobe,
sfprobe and tee.
+ Introduced bgp_table_dump_latest_file: defines the full pathname to
pointer(s) to latest file(s). Update of the latest pointer is done
evaluating files modification time. Many thanks to Juan Camilo Cardona
( @jccardonar ) for proposing the feature.
+ Introduced pmacctd_nonroot config directive to allow to run pmacctd
from a user with non root privileges. This can be desirable on systems
supporting a tool like setcap, ie. 'setcap "cap_net_raw,cap_net_admin=ep"
/path/to/pmacctd', to assign specific system capabilities to unprivileged
users. Patch is courtesy by Laurent Oudot ( @loudot-tehtris ).
+ Introduced plugin_pipe_check_core_pid: when enabled (default), validates
the sender of data at the plugin side. Useful when plugin_pipe_amqp or
plugin_pipe_kafka are enabled and hence a broker sits between the daemon
Core Process and the Plugins.
+ A new debug_internal_msg config directive to specifically enable debug
of internal messaging between Core process and plugins.
! bgp_table_dump_refresh_time, bmp_dump_refresh_time: max allowed value
raised to 86400 from 3600.
! [n|s]facctd_as_new renamed [n|s]facctd_as; improved input checks to all
*_as (ie. nfacctd_as) and *_net (ie. nfacctd_net) config directives.
! pkt_handlers.c: NF_sampling_rate_handler(), SF_sampling_rate_handler()
now perform a renormalization check at last (instead of at first) so to
report the case of unknown (0) sampling rate.
! plugin_pipe_amqp_routing_key: default value changed to '$core_proc_name-
$plugin_name-$plugin_type'. Also, increased flexibility for customizing
the key with the use of variables (values computed at startup).
! Improved amqp_receiver.py example with CL arguments and better exception
handling. Also removed file amqp_receiver_trace.py, example is now merged
in amqp_receiver.py.
! fix, BMP daemon: greatly improved message parsing and segment reassembly;
RabbitMQ broker support found broken; several code optimizations are also
included.
! fix, plugin_pipe_amqp_routing_key: check introduced to prevent multiple
plugins to bind to the same RabbitMQ exchange, routing key combination.
Thanks to Jerred Horsman for reporting the issue.
! fix, MongoDB plugin: added a custom oid fuzz generator to prevent
concurrent inserts to fail; switched from deprecated mongo_connect() to
mongo_client(); added MONGO_CONTINUE_ON_ERROR flag to mongo_insert_batch
along with more verbose error reporting. Patches are all courtesy by
Russell Heilling ( @xchewtoyx ).
! fix, nl.c: increments made too early after introduction of MAX_GTP_TRIALS
Affected: pmacctd processing of GTP in releases 1.5.x. Patch is courtesy
by TANAKA Masayuki ( @tanakamasayuki ).
! fix, pkt_handlers.c: improved case for no SAMPLER_ID, ALU & IPFIX in
NF_sampling_rate_handler() on par with NF_counters_renormalize_handler().
! fix, SQL scripts: always use "DROP TABLE IF EXISTS" for both PostgreSQL
and SQLite. Pathes are courtesy by Vincent Bernat ( @vincentbernat ).
! fix, plugin_hooks.c: if p_amqp_publish_binary() calls were done while a
sleeper thread was launched, a memory corruption was observed.
! fix, util.c: mkdir() calls in mkdir_multilevel() now default to mode 777
instead of 700; this allows more play with files_umask (by default 077).
Thanks to Ruben Laban for reporting the issue.
! fix, BMP daemon: solved a build issue under MacOS X. Path is courtesy by
Junpei YOSHINO ( @junpei-yoshino ).
! fix, util.c: self-defined Malloc() can allocate more than 4GB of memory;
function is also now renamed pm_malloc().
! fix, PostgreSQL plugin: upon purge, call sql_query() only if status of
the entry is SQL_CACHE_COMMITTED. Thanks to Harry Foster ( @harryfoster )
for his support resolving the issue.
! fix, building system: link pfring before pcap to prevend failures when
linking. Patch is courtesy by @matthewsf .
! fix, plugin_common.c: memory leak discovered when pending queries queue
was involved (ie. cases where print_refresh_time > print_history). Thanks
to Edward Henigin for reporting the issue.
-------------------------------------------------------------------
Tue Sep 8 15:11:04 UTC 2015 - mardnh@gmx.de
- update to version 1.5.2
- add patch: pmacct-fix-implicit-pointer-decl.diff
-------------------------------------------------------------------
Sun Jul 26 07:57:48 UTC 2015 - mardnh@gmx.de
- do not build with ULOG on newer versions > 13.2 since it got removed
from mainstream linux kernel >= 3.17
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=7200135bc1e61f1437dc326ae2ef2f310c50b4eb
-------------------------------------------------------------------
Sat Feb 21 20:28:55 UTC 2015 - mardnh@gmx.de
- update to version 1.5.1
+ BMP daemon: BMP, BGP Monitoring Protocol, can be used to monitor BGP
sessions. The current implementation is base on the draft-ietf-grow-bmp-07
IETF draft. The daemon currently supports BMP events and stats only, ie.
initiation, termination, peer up, peer down and stats reports messages.
Route Monitoring is future (upcoming) work but routes can be currently
sourced via the BGP daemon thread (best path only or ADD-PATH), making
the two daemons complementary. The daemon enables to write BMP messages
to files or AMQP queues, real-time (msglog) or at regular time intervals
(dump) and is a separate thread in the NetFlow (nfacctd) or sFlow (sfacctd)
collectors.
+ tmp_net_own_field directive is introduced to record both individual source
and destination IP addresses and their IP prefix (nets) as part of the same
aggregation method. While this should become default behaviour, a knob for
backward-compatibility is made available for all 1.5 until the next major
release.
+ Introduced nfacctd_stitching and equivalents (ie. sfacctd_stitching):
when set to true, given an aggregation method, two new non-key fields are
added to the aggregate upon purging data to the backend: timestamp_min is
the timestamp of the first element contributing to a certain aggregate
and timestamp_max is the timestamp of the last element. In case the export
protocol provides time references, ie. NetFlow/IPFIX, these are used; if not
the current time (hence time of arrival to the collector) is used instead.
+ Introduced amqp_routing_key_rr feature to perform round-robin load-
balancing over a set of routing keys. This is in addition to existing,
and more involved, functionality of tag-based load-balancing.
+ Introduced amqp_multi_values feature: this is same feature in concept as
sql_multi_values (see docs). The value is the amount of elements to pack
in each JSON array.
+ Introduced amqp_vhost and companion (ie. bgp_daemon_msglog_amqp_vhost)
configuration directives to define the AMQP/RabbitMQ server virtual host.
+ BGP daemon: bgp_daemon_id now allows to define the BGP Router-ID disjoint
from the bgp_daemon_ip definition. Thanks to Bela Toros for his patch.
+ tee plugin: introduced tee_ipprec feature to color replicated packets,
both in transparent and non-transparent modes. Useful, especially when
in transparent mode and replicating to hosts in different subnets, to
verify which packets are coming from the replicator.
+ tee plugin: plugin-kernel send buffer size is now configurable via a new
config directive tee_pipe_size. Improved logging of send() failures.
+ nfacctd: introduced support for IPFIX sampling/renormalization using
element IDs: #302 (selectorId), #305 (samplingPacketInterval) and #306
(samplingPacketSpace). Many thanks to Rene Stoutjesdijk for his support.
+ nfacctd: added also support for VLAN ID for NetFlow v9/IPFIX via element
type #243 (it was already supported via elements #58 and #59). Support was
also added for 802.1p/CoS via element #244.
+ nfacctd: added native support for NetFlow v9/IPFIX IE #252 and #253 as
part of existing primitives in_iface and out_iface (additional check).
+ pre_tag_map: introduced 'cvlan primitive. In NetFlow v9 and IPFIX this is
compared against IE #245. The primitive also supports map indexing.
+ Introduced pre_tag_label_filter to filter on the 'label' primitive in a
similar way how the existing pre_tag_filter feature works against the
'tag' primitive. Null label values (ie. unlabelled data) can be matched
using the 'null' keyword. Negations are allowed by pre-pending a minus
sign to the label value.
+ IMT plugin: introduced '-i' command-line option to pmacct client tool: it
shows last time (in seconds) statistis were cleared via 'pmacct -e'.
+ print, MongoDB & AMQP plugins: sql_startup_delay feature ported to these
plugins.
! sql_num_hosts: the feature has been improved to support IPv6 addresses.
Pre-requisite is definition of INET6_ATON() function in the RDBMS, which
is the case for MySQL >= 5.6.3. In SQLite such function has to be defined
manually.
! nfacctd: improved NF_evaluate_flow_type() euristics to reckon NetFlow/
IPFIX event (NAT, Firewall, etc.) vs traffic (flows) records.
! fix, GeoIP: spit log notification (warning) in case GeoIP_open() returns
null pointer.
! fix, IMT plugin: pmacct client -M and -N queries were failing to report
results on exact matches. Affected: 1.5.0. Thanks to Xavier Vitard for
reporting the issue.
! fix, pkt_handlers.c: missing else in NF_src_host_handler() was causing
IPv6 prefix being copied instead of IPv6 address against NetFlow v9 recs
containing both info.
! fix, uacctd: informational log message now shows the correct group the
daemon is bound to. Thanks to Marco Marzetti for reporting the issue.
! fix, nfv9_template.c: missing byte conversion while decoding templates
was causing SEGV under certain conditions. Thanks to Sergio Bellini for
reporting the issue.
-------------------------------------------------------------------
Thu Nov 6 21:50:04 UTC 2014 - mardnh@gmx.de
- temporary workaround for misc compile issues
* removed post-build-checks
-------------------------------------------------------------------
Wed Sep 17 13:50:20 UTC 2014 - mardnh@gmx.de
- update to version 1.5.0
- specfile cleanup
-------------------------------------------------------------------
Fri Jul 30 06:09:43 UTC 2010 - pascal.bleser@opensuse.org
- update to 0.12.3:
* a 'cos' aggregation primitive has been implemented, providing support for 802.1p priority
* TCP MD5 signatures are supported as part of the BGP daemon
* in nfprobe and sfprobe, the concept of traffic direction has been introduced, and as a result [ns]fprobe_direction and [ns]fprobe_ifindex config directives have been implemented
* Switch Extension Header support and Counter Samples for multiple interface features have been added in sfprobe
* a number of bugfixes are included
-------------------------------------------------------------------
Thu May 27 11:48:10 UTC 2010 - nix@opensuse.org
- update to 0.12.2
-------------------------------------------------------------------
Wed Feb 17 00:00:00 UTC 2010 - pascal.bleser@opensuse.org
- update to 0.12.0:
* the "is_symmetric" aggregation primitive has been implemented
and is aimed at easing detection of asymmetric traffic
* tagging is now possible on BGP primitives
* various fixes are also included
-------------------------------------------------------------------
Mon Jul 21 00:00:00 UTC 2008 - peter+rpmspam@suntel.com.tr
- Update to version 0.11.5
-------------------------------------------------------------------
Fri Nov 17 00:00:00 UTC 2006 - peter+rpmspam@suntel.com.tr
- Changed the permissions of the conf files to writable
-------------------------------------------------------------------
Tue May 16 00:00:00 UTC 2006 - peter+rpmspam@suntel.com.tr
- Cleaned up SPEC file a some more and updated to 0.10.1
-------------------------------------------------------------------
Wed May 10 00:00:00 UTC 2006 - peter+rpmspam@suntel.com.tr
- Fixup spec file to work properly on SUSE Build system
-------------------------------------------------------------------
Tue Dec 27 00:00:00 UTC 2005 - silfreed@silfreed.net
- upgraded to 0.9.6
- split database backends into separate binaries
- added 64bit counter support by default
-------------------------------------------------------------------
Thu Dec 8 00:00:00 UTC 2005 - silfreed@silfreed.net
- upgraded to 0.9.5
- removed shortver
-------------------------------------------------------------------
Sun Nov 13 00:00:00 UTC 2005 - silfreed@silfreed.net
- upgraded to 0.9.4
-------------------------------------------------------------------
Wed Oct 12 00:00:00 UTC 2005 - silfreed@silfreed.net
- upgraded to 0.9.3
-------------------------------------------------------------------
Tue Aug 23 00:00:00 UTC 2005 - silfreed@silfreed.net
- upgraded to 0.9.1
-------------------------------------------------------------------
Tue May 24 00:00:00 UTC 2005 - silfreed@silfreed.net
- changed 'libmysqlclient' Require to 'mysql'
-------------------------------------------------------------------
Mon May 23 00:00:00 UTC 2005 - silfreed@silfreed.net
- removed epoch (old versions won't upgrade correctly)
- upgraded to 0.8.6
-------------------------------------------------------------------
Sat Apr 10 00:00:00 UTC 2004 - dwarner@ctinetworks.com
- Initial RPM release.
Reference in New Issue View Git Blame Copy Permalink