Re: stmmac/RTL8211F/Meson GXBB: TX throughput problems

From: "André Roth" <neolynx@gmail•com>
To: Martin Blumenstingl <martin.blumenstingl@googlemail•com>
Cc: Alexandre Torgue <alexandre.torgue@st•com>,
	Johnson Leung <r58129@freescale•com>,
	netdev@vger•kernel.org,
	Giuseppe Cavallaro <peppe.cavallaro@st•com>,
	linux-amlogic@lists•infradead.org
Subject: Re: stmmac/RTL8211F/Meson GXBB: TX throughput problems
Date: Sat, 17 Sep 2016 23:23:12 +0200	[thread overview]
Message-ID: <20160917232312.1e30d425@gmail.com> (raw)
In-Reply-To: <CAFBinCD42A_SqQz8GM-H+A32dMELknAytHuJ5T2YgZ3BrMKA3A@mail.gmail.com>

[-- Attachment #1: Type: text/plain, Size: 3257 bytes --]

Hi all,

I have an odroid c2 board which shows this issue. No data is
transmitted or received after a moment of intense tx traffic. Copying a
1GB file per scp from the board triggers it repeatedly. 

The board has a stmmac - user ID: 0x11, Synopsys ID: 0x37.

When switching the network to 100Mb/s the copying does
not seam to trigger the issue.

I've attached the ethtool statistics before and after the problem.

Thanks for your help, 

 André

> Hi Alexandre,
> 
> On Mon, Sep 12, 2016 at 6:37 PM, Alexandre Torgue
> <alexandre.torgue@st•com> wrote:
> > Which Synopsys IP version do you use ?  
> found this in a dmesg log:
> [    1.504784] stmmac - user ID: 0x11, Synopsys ID: 0x37
> [    1.509785]  Ring mode enabled
> [    1.512796]  DMA HW capability register supported
> [    1.517286]  Normal descriptors
> [    1.520565]  RX Checksum Offload Engine supported
> [    1.525219]  COE Type 2
> [    1.527638]  TX Checksum insertion supported
> [    1.531862]  Wake-Up On Lan supported
> [    1.535483]  Enable RX Mitigation via HW Watchdog Timer
> [    1.543851] libphy: stmmac: probed
> [    1.544025] eth0: PHY ID 001cc916 at 0 IRQ POLL (stmmac-0:00)
> active [    1.550321] eth0: PHY ID 001cc916 at 7 IRQ POLL
> (stmmac-0:07)
> 
> >> Gbit ethernet on my device is provided by a Realtek RTL8211F RGMII
> >> PHY. Similar issues were reported in #linux-amlogic by a user with
> >> an Odroid C2 board (= similar hardware).
> >>
> >> The symptoms are:
> >> Receiving data is plenty fast (I can max out my internet connection
> >> easily, and with iperf3 I get ~900Mbit/s).
> >> Transmitting data from the device is unfortunately very slow,
> >> traffic sometimes even stalls completely.
> >>
> >> I have attached the iperf results and the output of
> >> /sys/kernel/debug/stmmaceth/eth0/descriptors_status.
> >> Below you can find the ifconfig, netstat and stmmac dma_cap info
> >> (*after* I ran all tests).
> >>
> >> The "involved parties" are:
> >> - Meson GXBB specific network configuration registers (I have have
> >> double-checked them with the reference drivers: everything seems
> >> fine here)
> >> - stmmac: it seems that nobody else has reported these kind of
> >> issues so far, however I'd still like to hear where I should
> >> enable some debugging bits to rule out any stmmac bug  
> >
> >
> > On my side, I just tested on the same "kind" of system:
> > -SYNOPSYS GMAC 3.7
> > -RTL8211EG as PHY
> >
> > With I perf, I reach:
> >         -RX: 932 Mbps
> >         -TX: 820Mbps
> >
> > Can you check ethtool -S eth0 (most precisely "MMC"counter and
> > errors) ? Which kernel version do you use ?  
> I am using a 4.8.0-rc4 kernel, based on Kevin's "integration" branch:
> [0] Unfortunately I don't have access to my device in the next few
> days, but I'll keep you updated once I have the ethtool output.
> 
> 
> Thanks for your time
> Regards,
> Martin
> 
> 
> [0]
> https://git.kernel.org/cgit/linux/kernel/git/khilman/linux-amlogic.git/log/?h=v4.8/integ
> 
> _______________________________________________
> linux-amlogic mailing list
> linux-amlogic@lists•infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-amlogic
> 

[-- Attachment #2: ethstats.after --]
[-- Type: application/octet-stream, Size: 5155 bytes --]

NIC statistics:
     mmc_tx_octetcount_gb: 0
     mmc_tx_framecount_gb: 0
     mmc_tx_broadcastframe_g: 0
     mmc_tx_multicastframe_g: 0
     mmc_tx_64_octets_gb: 0
     mmc_tx_65_to_127_octets_gb: 0
     mmc_tx_128_to_255_octets_gb: 0
     mmc_tx_256_to_511_octets_gb: 0
     mmc_tx_512_to_1023_octets_gb: 0
     mmc_tx_1024_to_max_octets_gb: 0
     mmc_tx_unicast_gb: 0
     mmc_tx_multicast_gb: 0
     mmc_tx_broadcast_gb: 0
     mmc_tx_underflow_error: 0
     mmc_tx_singlecol_g: 0
     mmc_tx_multicol_g: 0
     mmc_tx_deferred: 0
     mmc_tx_latecol: 0
     mmc_tx_exesscol: 0
     mmc_tx_carrier_error: 0
     mmc_tx_octetcount_g: 0
     mmc_tx_framecount_g: 0
     mmc_tx_excessdef: 0
     mmc_tx_pause_frame: 0
     mmc_tx_vlan_frame_g: 0
     mmc_rx_framecount_gb: 31723
     mmc_rx_octetcount_gb: 2360579
     mmc_rx_octetcount_g: 2360579
     mmc_rx_broadcastframe_g: 0
     mmc_rx_multicastframe_g: 0
     mmc_rx_crc_error: 0
     mmc_rx_align_error: 0
     mmc_rx_run_error: 0
     mmc_rx_jabber_error: 0
     mmc_rx_undersize_g: 0
     mmc_rx_oversize_g: 0
     mmc_rx_64_octets_gb: 4
     mmc_rx_65_to_127_octets_gb: 31700
     mmc_rx_128_to_255_octets_gb: 11
     mmc_rx_256_to_511_octets_gb: 2
     mmc_rx_512_to_1023_octets_gb: 5
     mmc_rx_1024_to_max_octets_gb: 1
     mmc_rx_unicast_g: 31723
     mmc_rx_length_error: 0
     mmc_rx_autofrangetype: 0
     mmc_rx_pause_frames: 0
     mmc_rx_fifo_overflow: 0
     mmc_rx_vlan_frames_gb: 0
     mmc_rx_watchdog_error: 0
     mmc_rx_ipc_intr_mask: 2147385342
     mmc_rx_ipc_intr: 0
     mmc_rx_ipv4_gd: 31719
     mmc_rx_ipv4_hderr: 0
     mmc_rx_ipv4_nopay: 0
     mmc_rx_ipv4_frag: 0
     mmc_rx_ipv4_udsbl: 0
     mmc_rx_ipv4_gd_octets: 1789381
     mmc_rx_ipv4_hderr_octets: 0
     mmc_rx_ipv4_nopay_octets: 0
     mmc_rx_ipv4_frag_octets: 0
     mmc_rx_ipv4_udsbl_octets: 0
     mmc_rx_ipv6_gd_octets: 0
     mmc_rx_ipv6_hderr_octets: 0
     mmc_rx_ipv6_nopay_octets: 0
     mmc_rx_ipv6_gd: 0
     mmc_rx_ipv6_hderr: 0
     mmc_rx_ipv6_nopay: 0
     mmc_rx_udp_gd: 27
     mmc_rx_udp_err: 0
     mmc_rx_tcp_gd: 31692
     mmc_rx_tcp_err: 0
     mmc_rx_icmp_gd: 0
     mmc_rx_icmp_err: 0
     mmc_rx_udp_gd_octets: 3856
     mmc_rx_udp_err_octets: 0
     mmc_rx_tcp_gd_octets: 1151145
     mmc_rx_tcp_err_octets: 0
     mmc_rx_icmp_gd_octets: 0
     mmc_rx_icmp_err_octets: 0
     tx_underflow: 0
     tx_carrier: 0
     tx_losscarrier: 0
     vlan_tag: 0
     tx_deferred: 0
     tx_vlan: 0
     tx_jabber: 0
     tx_frame_flushed: 0
     tx_payload_error: 0
     tx_ip_header_error: 0
     rx_desc: 0
     sa_filter_fail: 0
     overflow_error: 0
     ipc_csum_error: 0
     rx_collision: 0
     rx_crc: 0
     dribbling_bit: 0
     rx_length: 0
     rx_mii: 0
     rx_multicast: 0
     rx_gmac_overflow: 0
     rx_watchdog: 0
     da_rx_filter_fail: 0
     sa_rx_filter_fail: 0
     rx_missed_cntr: 0
     rx_overflow_cntr: 0
     rx_vlan: 0
     tx_undeflow_irq: 0
     tx_process_stopped_irq: 0
     tx_jabber_irq: 0
     rx_overflow_irq: 0
     rx_buf_unav_irq: 0
     rx_process_stopped_irq: 0
     rx_watchdog_irq: 0
     tx_early_irq: 0
     fatal_bus_error_irq: 0
     rx_early_irq: 2931
     threshold: 1
     tx_pkt_n: 307436
     rx_pkt_n: 31706
     normal_irq_n: 24352
     rx_normal_irq_n: 15239
     napi_poll: 24346
     tx_normal_irq_n: 9161
     tx_clean: 24463
     tx_set_ic_bit: 9366
     irq_receive_pmt_irq_n: 0
     mmc_tx_irq_n: 0
     mmc_rx_irq_n: 0
     mmc_rx_csum_offload_irq_n: 0
     irq_tx_path_in_lpi_mode_n: 12723
     irq_tx_path_exit_lpi_mode_n: 12722
     irq_rx_path_in_lpi_mode_n: 0
     irq_rx_path_exit_lpi_mode_n: 0
     phy_eee_wakeup_error_n: 0
     ip_hdr_err: 0
     ip_payload_err: 0
     ip_csum_bypassed: 0
     ipv4_pkt_rcvd: 0
     ipv6_pkt_rcvd: 0
     rx_msg_type_ext_no_ptp: 0
     rx_msg_type_sync: 0
     rx_msg_type_follow_up: 0
     rx_msg_type_delay_req: 0
     rx_msg_type_delay_resp: 0
     rx_msg_type_pdelay_req: 0
     rx_msg_type_pdelay_resp: 0
     rx_msg_type_pdelay_follow_up: 0
     ptp_frame_type: 0
     ptp_ver: 0
     timestamp_dropped: 0
     av_pkt_rcvd: 0
     av_tagged_pkt_rcvd: 0
     vlan_tag_priority_val: 0
     l3_filter_match: 0
     l4_filter_match: 0
     l3_l4_filter_no_match: 0
     irq_pcs_ane_n: 0
     irq_pcs_link_n: 0
     irq_rgmii_n: 1
     mtl_tx_status_fifo_full: 0
     mtl_tx_fifo_not_empty: 0
     mmtl_fifo_ctrl: 0
     mtl_tx_fifo_read_ctrl_write: 0
     mtl_tx_fifo_read_ctrl_wait: 0
     mtl_tx_fifo_read_ctrl_read: 0
     mtl_tx_fifo_read_ctrl_idle: 0
     mac_tx_in_pause: 0
     mac_tx_frame_ctrl_xfer: 0
     mac_tx_frame_ctrl_idle: 0
     mac_tx_frame_ctrl_wait: 0
     mac_tx_frame_ctrl_pause: 0
     mac_gmii_tx_proto_engine: 0
     mtl_rx_fifo_fill_level_full: 0
     mtl_rx_fifo_fill_above_thresh: 0
     mtl_rx_fifo_fill_below_thresh: 0
     mtl_rx_fifo_fill_level_empty: 0
     mtl_rx_fifo_read_ctrl_flush: 0
     mtl_rx_fifo_read_ctrl_read_data: 0
     mtl_rx_fifo_read_ctrl_status: 0
     mtl_rx_fifo_read_ctrl_idle: 0
     mtl_rx_fifo_ctrl_active: 0
     mac_rx_frame_ctrl_fifo: 0
     mac_gmii_rx_proto_engine: 0
     tx_tso_frames: 0
     tx_tso_nfrags: 0

[-- Attachment #3: ethstats.before --]
[-- Type: application/octet-stream, Size: 5093 bytes --]

NIC statistics:
     mmc_tx_octetcount_gb: 0
     mmc_tx_framecount_gb: 0
     mmc_tx_broadcastframe_g: 0
     mmc_tx_multicastframe_g: 0
     mmc_tx_64_octets_gb: 0
     mmc_tx_65_to_127_octets_gb: 0
     mmc_tx_128_to_255_octets_gb: 0
     mmc_tx_256_to_511_octets_gb: 0
     mmc_tx_512_to_1023_octets_gb: 0
     mmc_tx_1024_to_max_octets_gb: 0
     mmc_tx_unicast_gb: 0
     mmc_tx_multicast_gb: 0
     mmc_tx_broadcast_gb: 0
     mmc_tx_underflow_error: 0
     mmc_tx_singlecol_g: 0
     mmc_tx_multicol_g: 0
     mmc_tx_deferred: 0
     mmc_tx_latecol: 0
     mmc_tx_exesscol: 0
     mmc_tx_carrier_error: 0
     mmc_tx_octetcount_g: 0
     mmc_tx_framecount_g: 0
     mmc_tx_excessdef: 0
     mmc_tx_pause_frame: 0
     mmc_tx_vlan_frame_g: 0
     mmc_rx_framecount_gb: 30
     mmc_rx_octetcount_gb: 5049
     mmc_rx_octetcount_g: 5049
     mmc_rx_broadcastframe_g: 0
     mmc_rx_multicastframe_g: 0
     mmc_rx_crc_error: 0
     mmc_rx_align_error: 0
     mmc_rx_run_error: 0
     mmc_rx_jabber_error: 0
     mmc_rx_undersize_g: 0
     mmc_rx_oversize_g: 0
     mmc_rx_64_octets_gb: 4
     mmc_rx_65_to_127_octets_gb: 13
     mmc_rx_128_to_255_octets_gb: 8
     mmc_rx_256_to_511_octets_gb: 1
     mmc_rx_512_to_1023_octets_gb: 4
     mmc_rx_1024_to_max_octets_gb: 0
     mmc_rx_unicast_g: 30
     mmc_rx_length_error: 0
     mmc_rx_autofrangetype: 0
     mmc_rx_pause_frames: 0
     mmc_rx_fifo_overflow: 0
     mmc_rx_vlan_frames_gb: 0
     mmc_rx_watchdog_error: 0
     mmc_rx_ipc_intr_mask: 1073692671
     mmc_rx_ipc_intr: 0
     mmc_rx_ipv4_gd: 26
     mmc_rx_ipv4_hderr: 0
     mmc_rx_ipv4_nopay: 0
     mmc_rx_ipv4_frag: 0
     mmc_rx_ipv4_udsbl: 0
     mmc_rx_ipv4_gd_octets: 4325
     mmc_rx_ipv4_hderr_octets: 0
     mmc_rx_ipv4_nopay_octets: 0
     mmc_rx_ipv4_frag_octets: 0
     mmc_rx_ipv4_udsbl_octets: 0
     mmc_rx_ipv6_gd_octets: 0
     mmc_rx_ipv6_hderr_octets: 0
     mmc_rx_ipv6_nopay_octets: 0
     mmc_rx_ipv6_gd: 0
     mmc_rx_ipv6_hderr: 0
     mmc_rx_ipv6_nopay: 0
     mmc_rx_udp_gd: 26
     mmc_rx_udp_err: 0
     mmc_rx_tcp_gd: 0
     mmc_rx_tcp_err: 0
     mmc_rx_icmp_gd: 0
     mmc_rx_icmp_err: 0
     mmc_rx_udp_gd_octets: 3805
     mmc_rx_udp_err_octets: 0
     mmc_rx_tcp_gd_octets: 0
     mmc_rx_tcp_err_octets: 0
     mmc_rx_icmp_gd_octets: 0
     mmc_rx_icmp_err_octets: 0
     tx_underflow: 0
     tx_carrier: 0
     tx_losscarrier: 0
     vlan_tag: 0
     tx_deferred: 0
     tx_vlan: 0
     tx_jabber: 0
     tx_frame_flushed: 0
     tx_payload_error: 0
     tx_ip_header_error: 0
     rx_desc: 0
     sa_filter_fail: 0
     overflow_error: 0
     ipc_csum_error: 0
     rx_collision: 0
     rx_crc: 0
     dribbling_bit: 0
     rx_length: 0
     rx_mii: 0
     rx_multicast: 0
     rx_gmac_overflow: 0
     rx_watchdog: 0
     da_rx_filter_fail: 0
     sa_rx_filter_fail: 0
     rx_missed_cntr: 0
     rx_overflow_cntr: 0
     rx_vlan: 0
     tx_undeflow_irq: 0
     tx_process_stopped_irq: 0
     tx_jabber_irq: 0
     rx_overflow_irq: 0
     rx_buf_unav_irq: 0
     rx_process_stopped_irq: 0
     rx_watchdog_irq: 0
     tx_early_irq: 0
     fatal_bus_error_irq: 0
     rx_early_irq: 0
     threshold: 1
     tx_pkt_n: 162
     rx_pkt_n: 30
     normal_irq_n: 31
     rx_normal_irq_n: 29
     napi_poll: 31
     tx_normal_irq_n: 2
     tx_clean: 138
     tx_set_ic_bit: 2
     irq_receive_pmt_irq_n: 0
     mmc_tx_irq_n: 0
     mmc_rx_irq_n: 0
     mmc_rx_csum_offload_irq_n: 0
     irq_tx_path_in_lpi_mode_n: 124
     irq_tx_path_exit_lpi_mode_n: 123
     irq_rx_path_in_lpi_mode_n: 0
     irq_rx_path_exit_lpi_mode_n: 0
     phy_eee_wakeup_error_n: 0
     ip_hdr_err: 0
     ip_payload_err: 0
     ip_csum_bypassed: 0
     ipv4_pkt_rcvd: 0
     ipv6_pkt_rcvd: 0
     rx_msg_type_ext_no_ptp: 0
     rx_msg_type_sync: 0
     rx_msg_type_follow_up: 0
     rx_msg_type_delay_req: 0
     rx_msg_type_delay_resp: 0
     rx_msg_type_pdelay_req: 0
     rx_msg_type_pdelay_resp: 0
     rx_msg_type_pdelay_follow_up: 0
     ptp_frame_type: 0
     ptp_ver: 0
     timestamp_dropped: 0
     av_pkt_rcvd: 0
     av_tagged_pkt_rcvd: 0
     vlan_tag_priority_val: 0
     l3_filter_match: 0
     l4_filter_match: 0
     l3_l4_filter_no_match: 0
     irq_pcs_ane_n: 0
     irq_pcs_link_n: 0
     irq_rgmii_n: 1
     mtl_tx_status_fifo_full: 0
     mtl_tx_fifo_not_empty: 0
     mmtl_fifo_ctrl: 0
     mtl_tx_fifo_read_ctrl_write: 0
     mtl_tx_fifo_read_ctrl_wait: 0
     mtl_tx_fifo_read_ctrl_read: 0
     mtl_tx_fifo_read_ctrl_idle: 0
     mac_tx_in_pause: 0
     mac_tx_frame_ctrl_xfer: 0
     mac_tx_frame_ctrl_idle: 0
     mac_tx_frame_ctrl_wait: 0
     mac_tx_frame_ctrl_pause: 0
     mac_gmii_tx_proto_engine: 0
     mtl_rx_fifo_fill_level_full: 0
     mtl_rx_fifo_fill_above_thresh: 0
     mtl_rx_fifo_fill_below_thresh: 0
     mtl_rx_fifo_fill_level_empty: 0
     mtl_rx_fifo_read_ctrl_flush: 0
     mtl_rx_fifo_read_ctrl_read_data: 0
     mtl_rx_fifo_read_ctrl_status: 0
     mtl_rx_fifo_read_ctrl_idle: 0
     mtl_rx_fifo_ctrl_active: 0
     mac_rx_frame_ctrl_fifo: 0
     mac_gmii_rx_proto_engine: 0
     tx_tso_frames: 0
     tx_tso_nfrags: 0