Syncd和SAI
The [Syncd container](. /2-3-key-containers.html#asic management container syncd) is a container in SONiC dedicated to managing ASICs, where the core process syncd is responsible for communicating with the Redis database, loading SAIs and interacting with them to complete ASIC initialization, configuration and status reporting processing, and so on.
Since a large number of workflows in SONiC end up needing to interact with ASIC through Syncd and SAI, this part becomes a common part of these workflows, so let's take a look at how Syncd and SAI work before we expand on other workflows.
Syncd启动流程
The entry point of the syncd process is in the syncd_main.cpp function, and the overall process of its startup is roughly divided into two parts.
The first part is the creation of individual objects and their initialization:
sequenceDiagram
autonumber
participant SDM as syncd_main
participant SD as Syncd
participant SAI as VendorSai
SDM->>+SD: 调用构造函数
SD->>SD: 加载和解析命令行参数和配置文件
SD->>SD: 创建数据库相关对象，如：<br/>ASIC_DB Connector和FlexCounterManager
SD->>SD: 创建MDIO IPC服务器
SD->>SD: 创建SAI上报处理逻辑
SD->>SD: 创建RedisSelectableChannel用于接收Redis通知
SD->>-SAI: 初始化SAI
The second part starts the main loop and handles the initialization events:
sequenceDiagram
autonumber
box purple 主线程
participant SDM as syncd_main
participant SD as Syncd
participant SAI as VendorSai
end
box darkblue 通知处理线程
participant NP as NotificationProcessor
end
box darkgreen MDIO IPC服务器线程
participant MIS as MdioIpcServer
end
SDM->>+SD: 启动主线程循环
SD->>NP: 启动SAI上报处理线程
NP->>NP: 开始通知处理循环
SD->>MIS: 启动MDIO IPC服务器线程
MIS->>MIS: 开始MDIO IPC服务器事件循环
SD->>SD: 初始化并启动事件分发机制，开始主循环
loop 处理事件
alt 如果是创建Switch的事件或者是WarmBoot
SD->>SAI: 创建Switch对象，设置通知回调
else 如果是其他事件
SD->>SD: 处理事件
end
end
SD->>-SDM: 退出主循环返回
Then let's take a closer look at the process from a code perspective.
syncd_main函数
The syncd_main function itself is very simple, the main logic is to create the Syncd object and then call its run method:
// File: src/sonic-sairedis/syncd/syncd_main.cpp
int syncd_main(int argc, char **argv)
{
auto vendorSai = std::make_shared<VendorSai>();
auto syncd = std::make_shared<Syncd>(vendorSai, commandLineOptions, isWarmStart);
syncd->run();
return EXIT_SUCCESS;
}
The constructor of the Syncd object is responsible for initializing the various functions in Syncd, while the run method is responsible for starting the main loop of Syncd.
Syncd构造函数
The constructor of the Syncd object is responsible for creating or initializing various functions in Syncd, such as objects for connecting to the database, statistics management, and ASIC notification processing logic, etc. The main code is as follows:
// File: src/sonic-sairedis/syncd/Syncd.cpp
Syncd::Syncd(
_In_ std::shared_ptr<sairedis::SaiInterface> vendorSai,
_In_ std::shared_ptr<CommandLineOptions> cmd,
_In_ bool isWarmStart):
m_vendorSai(vendorSai),
...
{
...
// Load context config
auto ccc = sairedis::ContextConfigContainer::loadFromFile(m_commandLineOptions->m_contextConfig.c_str());
m_contextConfig = ccc->get(m_commandLineOptions->m_globalContext);
...
// Create FlexCounter manager
m_manager = std::make_shared<FlexCounterManager>(m_vendorSai, m_contextConfig->m_dbCounters);
// Create DB related objects
m_dbAsic = std::make_shared<swss::DBConnector>(m_contextConfig->m_dbAsic, 0);
m_mdioIpcServer = std::make_shared<MdioIpcServer>(m_vendorSai, m_commandLineOptions->m_globalContext);
m_selectableChannel = std::make_shared<sairedis::RedisSelectableChannel>(m_dbAsic, ASIC_STATE_TABLE, REDIS_TABLE_GETRESPONSE, TEMP_PREFIX, modifyRedis);
// Create notification processor and handler
m_notifications = std::make_shared<RedisNotificationProducer>(m_contextConfig->m_dbAsic);
m_client = std::make_shared<RedisClient>(m_dbAsic);
m_processor = std::make_shared<NotificationProcessor>(m_notifications, m_client, std::bind(&Syncd::syncProcessNotification, this, _1));
m_handler = std::make_shared<NotificationHandler>(m_processor);
m_sn.onFdbEvent = std::bind(&NotificationHandler::onFdbEvent, m_handler.get(), _1, _2);
m_sn.onNatEvent = std::bind(&NotificationHandler::onNatEvent, m_handler.get(), _1, _2);
// Init many other event handlers here
m_handler->setSwitchNotifications(m_sn.getSwitchNotifications());
...
// Initialize SAI
sai_status_t status = vendorSai->initialize(0, &m_test_services);
...
}
SAI的初始化与VendorSai
The last and most important step in the initialization of Syncd is the initialization of the SAI. [In the SAI introduction for the core component, we briefly showed the initialization of SAI, the implementation, and how it provides support for different platforms for SONiC](. /2-4-sai-intro.html), so here we will mainly look at how Syncd wraps and calls SAI.
Syncd uses VendorSai to encapsulate all the APIs of SAI, making it easy for the upper layers to call. Its initialization process is also very straightforward, basically a direct call to the above two functions and error handling, as follows:
// File: src/sonic-sairedis/syncd/VendorSai.cpp
sai_status_t VendorSai::initialize(
_In_ uint64_t flags,
_In_ const sai_service_method_table_t *service_method_table)
{
...
// Initialize SAI
memcpy(&m_service_method_table, service_method_table, sizeof(m_service_method_table));
auto status = sai_api_initialize(flags, service_method_table);
// If SAI is initialized successfully, query all SAI API methods.
// sai_metadata_api_query will also update all extern global sai_*_api variables, so we can also use
// sai_metadata_get_object_type_info to get methods for a specific SAI object type.
if (status == SAI_STATUS_SUCCESS) {
memset(&m_apis, 0, sizeof(m_apis));
int failed = sai_metadata_apis_query(sai_api_query, &m_apis);
...
}
...
return status;
}
Once all the SAI APIs are obtained, we can call the SAI APIs through the VendorSai object. Currently there are two main ways to call the SAI API.
The first one is called by sai_object_type_into_t, which is similar to implementing a dummy table for all SAI Objects, as follows:
// File: src/sonic-sairedis/syncd/VendorSai.cpp
sai_status_t VendorSai::set(
_In_ sai_object_type_t objectType,
_In_ sai_object_id_t objectId,
_In_ const sai_attribute_t *attr)
{
...
auto info = sai_metadata_get_object_type_info(objectType);
sai_object_meta_key_t mk = { .objecttype = objectType, .objectkey = { .key = { .object_id = objectId } } };
return info->set(&mk, attr);
}
The other way is to call through m_apis saved in the VendorSai object. This way is more direct, but before calling it, you need to call different APIs according to the type of SAI Object.
sai_status_t VendorSai::getStatsExt(
_In_ sai_object_type_t object_type,
_In_ sai_object_id_t object_id,
_In_ uint32_t number_of_counters,
_In_ const sai_stat_id_t *counter_ids,
_In_ sai_stats_mode_t mode,
_Out_ uint64_t *counters)
{
sai_status_t (*ptr)(
_In_ sai_object_id_t port_id,
_In_ uint32_t number_of_counters,
_In_ const sai_stat_id_t *counter_ids,
_In_ sai_stats_mode_t mode,
_Out_ uint64_t *counters);
switch ((int)object_type)
{
case SAI_OBJECT_TYPE_PORT:
ptr = m_apis.port_api->get_port_stats_ext;
break;
case SAI_OBJECT_TYPE_ROUTER_INTERFACE:
ptr = m_apis.router_interface_api->get_router_interface_stats_ext;
break;
case SAI_OBJECT_TYPE_POLICER:
ptr = m_apis.policer_api->get_policer_stats_ext;
break;
...
default:
SWSS_LOG_ERROR("not implemented, FIXME");
return SAI_STATUS_FAILURE;
}
return ptr(object_id, number_of_counters, counter_ids, mode, counters);
}
As you can clearly see, the code of the first call is much more concise and intuitive.
Syncd主循环
The main loop of Syncd is also using the standard [event distribution] in SONiC (. /4-3-event-polling-and-error-handling.html) mechanism: at startup, Syncd registers all Selectable objects used for event handling into the Select object used to fetch events, and then calls Select's select method in the main loop and wait for the event to occur. The core code is as follows:
// File: src/sonic-sairedis/syncd/Syncd.cpp
void Syncd::run()
{
volatile bool runMainLoop = true;
std::shared_ptr<swss::Select> s = std::make_shared<swss::Select>();
onSyncdStart(m_commandLineOptions->m_startType == SAI_START_TYPE_WARM_BOOT);
// Start notification processing thread
m_processor->startNotificationsProcessingThread();
// Start MDIO threads
for (auto& sw: m_switches) { m_mdioIpcServer->setSwitchId(sw.second->getRid()); }
m_mdioIpcServer->startMdioThread();
// Registering selectable for event polling
s->addSelectable(m_selectableChannel.get());
s->addSelectable(m_restartQuery.get());
s->addSelectable(m_flexCounter.get());
s->addSelectable(m_flexCounterGroup.get());
// Main event loop
while (runMainLoop)
{
swss::Selectable *sel = NULL;
int result = s->select(&sel);
...
if (sel == m_restartQuery.get()) {
// Handling switch restart event and restart switch here.
} else if (sel == m_flexCounter.get()) {
processFlexCounterEvent(*(swss::ConsumerTable*)sel);
} else if (sel == m_flexCounterGroup.get()) {
processFlexCounterGroupEvent(*(swss::ConsumerTable*)sel);
} else if (sel == m_selectableChannel.get()) {
// Handle redis updates here.
processEvent(*m_selectableChannel.get());
} else {
SWSS_LOG_ERROR("select failed: %d", result);
}
...
}
...
}
One of them, m_selectableChannel, is the object that is primarily responsible for handling events in the Redis database. It uses the [ProducerTable / ConsumerTable](. /4-2-2-redis-messaging-layer.md#producertable--consumertable) to interact with the Redis database, so all operations sent by the orchagent are stored in a list in Redis in the form of a triple, waiting for Syncd for processing. The core definition is as follows:
// File: src/sonic-sairedis/meta/RedisSelectableChannel.h
class RedisSelectableChannel: public SelectableChannel
{
public:
RedisSelectableChannel(
_In_ std::shared_ptr<swss::DBConnector> dbAsic,
_In_ const std::string& asicStateTable,
_In_ const std::string& getResponseTable,
_In_ const std::string& tempPrefix,
_In_ bool modifyRedis);
public: // SelectableChannel overrides
virtual bool empty() override;
...
public: // Selectable overrides
virtual int getFd() override;
virtual uint64_t readData() override;
...
private:
std::shared_ptr<swss::DBConnector> m_dbAsic;
std::shared_ptr<swss::ConsumerTable> m_asicState;
std::shared_ptr<swss::ProducerTable> m_getResponse;
...
};
In addition, when the main loop is started, Syncd starts two additional threads:
- 用于接收ASIC上报通知的通知处理线程：
m_processor->startNotificationsProcessingThread(); - 用于处理MDIO通信的MDIO IPC处理线程：
m_mdioIpcServer->startMdioThread();
We won't expand too much on their details in the initialization section, but will come back to them later when we introduce the relevant workflows.
创建Switch对象，初始化通知机制
After the main loop starts, Syncd will start calling the SAI API to create Switch objects. There are two entry points here, one is when ASIC_DB receives a notification to create a Switch, and the other is when Warm Boot, Syncd comes to initiate the call, but the internal flow of this step of creating a Switch is similar.
In the middle of this step, there is an important step to initialize the notification callbacks in the SAI's internal implementation to pass the notification handling logic we have created before to the SAI's implementation, such as FDB's events and so on. These callbacks will be passed as Attributes of the Switch to the SAI's create_switch method as parameters, and the SAI implementation will save them so that the callbacks can be called to notify Syncd when an event occurs. The core code here is as follows:
// File: src/sonic-sairedis/syncd/Syncd.cpp
sai_status_t Syncd::processQuadEvent(
_In_ sai_common_api_t api,
_In_ const swss::KeyOpFieldsValuesTuple &kco)
{
// Parse event into SAI object
sai_object_meta_key_t metaKey;
...
SaiAttributeList list(metaKey.objecttype, values, false);
sai_attribute_t *attr_list = list.get_attr_list();
uint32_t attr_count = list.get_attr_count();
// Update notifications pointers in attribute list
if (metaKey.objecttype == SAI_OBJECT_TYPE_SWITCH && (api == SAI_COMMON_API_CREATE || api == SAI_COMMON_API_SET))
{
m_handler->updateNotificationsPointers(attr_count, attr_list);
}
if (isInitViewMode())
{
// ProcessQuadEventInInitViewMode will eventually call into VendorSai, which calls create_swtich function in SAI.
sai_status_t status = processQuadEventInInitViewMode(metaKey.objecttype, strObjectId, api, attr_count, attr_list);
syncUpdateRedisQuadEvent(status, api, kco);
return status;
}
...
}
// File: src/sonic-sairedis/syncd/NotificationHandler.cpp
void NotificationHandler::updateNotificationsPointers(_In_ uint32_t attr_count, _In_ sai_attribute_t *attr_list) const
{
for (uint32_t index = 0; index < attr_count; ++index) {
...
sai_attribute_t &attr = attr_list[index];
switch (attr.id) {
...
case SAI_SWITCH_ATTR_SHUTDOWN_REQUEST_NOTIFY:
attr.value.ptr = (void*)m_switchNotifications.on_switch_shutdown_request;
break;
case SAI_SWITCH_ATTR_FDB_EVENT_NOTIFY:
attr.value.ptr = (void*)m_switchNotifications.on_fdb_event;
break;
...
}
...
}
}
// File: src/sonic-sairedis/syncd/Syncd.cpp
// Call stack: processQuadEvent
// -> processQuadEventInInitViewMode
// -> processQuadInInitViewModeCreate
// -> onSwitchCreateInInitViewMode
void Syncd::onSwitchCreateInInitViewMode(_In_ sai_object_id_t switchVid, _In_ uint32_t attr_count, _In_ const sai_attribute_t *attr_list)
{
if (m_switches.find(switchVid) == m_switches.end()) {
sai_object_id_t switchRid;
sai_status_t status;
status = m_vendorSai->create(SAI_OBJECT_TYPE_SWITCH, &switchRid, 0, attr_count, attr_list);
...
m_switches[switchVid] = std::make_shared<SaiSwitch>(switchVid, switchRid, m_client, m_translator, m_vendorSai);
m_mdioIpcServer->setSwitchId(switchRid);
...
}
...
}
From Mellanox's SAI implementation, we can see its specific approach to preservation:
static sai_status_t mlnx_create_switch(_Out_ sai_object_id_t * switch_id,
_In_ uint32_t attr_count,
_In_ const sai_attribute_t *attr_list)
{
...
status = find_attrib_in_list(attr_count, attr_list, SAI_SWITCH_ATTR_SWITCH_STATE_CHANGE_NOTIFY, &attr_val, &attr_idx);
if (!SAI_ERR(status)) {
g_notification_callbacks.on_switch_state_change = (sai_switch_state_change_notification_fn)attr_val->ptr;
}
status = find_attrib_in_list(attr_count, attr_list, SAI_SWITCH_ATTR_SHUTDOWN_REQUEST_NOTIFY, &attr_val, &attr_idx);
if (!SAI_ERR(status)) {
g_notification_callbacks.on_switch_shutdown_request =
(sai_switch_shutdown_request_notification_fn)attr_val->ptr;
}
status = find_attrib_in_list(attr_count, attr_list, SAI_SWITCH_ATTR_FDB_EVENT_NOTIFY, &attr_val, &attr_idx);
if (!SAI_ERR(status)) {
g_notification_callbacks.on_fdb_event = (sai_fdb_event_notification_fn)attr_val->ptr;
}
status = find_attrib_in_list(attr_count, attr_list, SAI_SWITCH_ATTR_PORT_STATE_CHANGE_NOTIFY, &attr_val, &attr_idx);
if (!SAI_ERR(status)) {
g_notification_callbacks.on_port_state_change = (sai_port_state_change_notification_fn)attr_val->ptr;
}
status = find_attrib_in_list(attr_count, attr_list, SAI_SWITCH_ATTR_PACKET_EVENT_NOTIFY, &attr_val, &attr_idx);
if (!SAI_ERR(status)) {
g_notification_callbacks.on_packet_event = (sai_packet_event_notification_fn)attr_val->ptr;
}
...
}
ASIC状态更新
ASIC status update is one of the most important workflows in Syncd, which is triggered when orchagent finds any change and starts modifying ASIC_DB to update ASIC via SAI. After understanding the main loop of Syncd, it is easy to understand the workflow of ASIC state update.
All steps occur in one thread in the main thread and are executed sequentially, summarized in a timing diagram as follows:
sequenceDiagram
autonumber
participant SD as Syncd
participant RSC as RedisSelectableChannel
participant SAI as VendorSai
participant R as Redis
loop 主线程循环
SD->>RSC: 收到epoll通知，通知获取所有到来的消息
RSC->>R: 通过ConsumerTable获取所有到来的消息
critical 给Syncd加锁
loop 所有收到的消息
SD->>RSC: 获取一个消息
SD->>SD: 解析消息，获取操作类型和操作对象
SD->>SAI: 调用对应的SAI API，更新ASIC
SD->>RSC: 发送调用结果给Redis
RSC->>R: 将调用结果写入Redis
end
end
end
First, the operation sent by orchagent through Redis is received by the RedisSelectableChannel object and then processed in the main loop. When Syncd processes to m_selectableChannel, it calls the processEvent method to process the operation. The core code for these steps has been described above when we introduced the main loop, so we won't go over it here.
Then, processEvent will call the corresponding SAI's API to update the ASIC according to the type of operation in it. The logic is a giant switch-case statement, as follows:
// File: src/sonic-sairedis/syncd/Syncd.cpp
void Syncd::processEvent(_In_ sairedis::SelectableChannel& consumer)
{
// Loop all operations in the queue
std::lock_guard<std::mutex> lock(m_mutex);
do {
swss::KeyOpFieldsValuesTuple kco;
consumer.pop(kco, isInitViewMode());
processSingleEvent(kco);
} while (!consumer.empty());
}
sai_status_t Syncd::processSingleEvent(_In_ const swss::KeyOpFieldsValuesTuple &kco)
{
auto& op = kfvOp(kco);
...
if (op == REDIS_ASIC_STATE_COMMAND_CREATE)
return processQuadEvent(SAI_COMMON_API_CREATE, kco);
if (op == REDIS_ASIC_STATE_COMMAND_REMOVE)
return processQuadEvent(SAI_COMMON_API_REMOVE, kco);
...
}
sai_status_t Syncd::processQuadEvent(
_In_ sai_common_api_t api,
_In_ const swss::KeyOpFieldsValuesTuple &kco)
{
// Parse operation
const std::string& key = kfvKey(kco);
const std::string& strObjectId = key.substr(key.find(":") + 1);
sai_object_meta_key_t metaKey;
sai_deserialize_object_meta_key(key, metaKey);
auto& values = kfvFieldsValues(kco);
SaiAttributeList list(metaKey.objecttype, values, false);
sai_attribute_t *attr_list = list.get_attr_list();
uint32_t attr_count = list.get_attr_count();
...
auto info = sai_metadata_get_object_type_info(metaKey.objecttype);
// Process the operation
sai_status_t status;
if (info->isnonobjectid) {
status = processEntry(metaKey, api, attr_count, attr_list);
} else {
status = processOid(metaKey.objecttype, strObjectId, api, attr_count, attr_list);
}
// Send response
if (api == SAI_COMMON_API_GET) {
sai_object_id_t switchVid = VidManager::switchIdQuery(metaKey.objectkey.key.object_id);
sendGetResponse(metaKey.objecttype, strObjectId, switchVid, status, attr_count, attr_list);
...
} else {
sendApiResponse(api, status);
}
syncUpdateRedisQuadEvent(status, api, kco);
return status;
}
sai_status_t Syncd::processEntry(_In_ sai_object_meta_key_t metaKey, _In_ sai_common_api_t api,
_In_ uint32_t attr_count, _In_ sai_attribute_t *attr_list)
{
...
switch (api)
{
case SAI_COMMON_API_CREATE:
return m_vendorSai->create(metaKey, SAI_NULL_OBJECT_ID, attr_count, attr_list);
case SAI_COMMON_API_REMOVE:
return m_vendorSai->remove(metaKey);
...
default:
SWSS_LOG_THROW("api %s not supported", sai_serialize_common_api(api).c_str());
}
}
ASIC状态变更上报
In turn, when any change in ASIC status occurs or data needs to be reported, it will also notify us via SAI, at which point Syncd will listen for these notifications and then report them to the orchagent via ASIC_DB. its main workflow is as follows:
sequenceDiagram
box purple SAI实现事件处理线程
participant SAI as SAI Impl
end
box darkblue 通知处理线程
participant NP as NotificationProcessor
participant SD as Syncd
participant RNP as RedisNotificationProducer
participant R as Redis
end
loop SAI实现事件处理消息循环
SAI->>SAI: 通过ASIC SDK获取事件
SAI->>SAI: 解析事件，并转换成SAI通知对象
SAI->>NP: 将通知对象序列化，<br/>并发送给通知处理线程的队列中
end
loop 通知处理线程消息循环
NP->>NP: 从队列中获取通知
NP->>SD: 获取Syncd锁
critical 给Syncd加锁
NP->>NP: 反序列化通知对象，并做一些处理
NP->>RNP: 重新序列化通知对象，并请求发送
RNP->>R: 将通知以NotificationProducer<br/>的形式写入ASIC_DB
end
end
Here we also look at the specific implementation. For a more in-depth understanding, we still analyze it with the help of the open source Mellanox SAI implementation.
At the very beginning, the SAI implementation needs to receive notifications from ASIC, this step is implemented through the ASIC SDK. Mellanox's SAI creates an event handling thread (event_thread) and then uses the select function to get and process the notifications sent from ASIC, the core code is as follows:
// File: platform/mellanox/mlnx-sai/SAI-Implementation/mlnx_sai/src/mlnx_sai_switch.c
static void event_thread_func(void *context)
{
#define MAX_PACKET_SIZE MAX(g_resource_limits.port_mtu_max, SX_HOST_EVENT_BUFFER_SIZE_MAX)
sx_status_t status;
sx_api_handle_t api_handle;
sx_user_channel_t port_channel, callback_channel;
fd_set descr_set;
int ret_val;
sai_object_id_t switch_id = (sai_object_id_t)context;
sai_port_oper_status_notification_t port_data;
sai_fdb_event_notification_data_t *fdb_events = NULL;
sai_attribute_t *attr_list = NULL;
...
// Init SDK API
if (SX_STATUS_SUCCESS != (status = sx_api_open(sai_log_cb, &api_handle))) {
if (g_notification_callbacks.on_switch_shutdown_request) {
g_notification_callbacks.on_switch_shutdown_request(switch_id);
}
return;
}
if (SX_STATUS_SUCCESS != (status = sx_api_host_ifc_open(api_handle, &port_channel.channel.fd))) {
goto out;
}
...
// Register for port and channel notifications
port_channel.type = SX_USER_CHANNEL_TYPE_FD;
if (SX_STATUS_SUCCESS != (status = sx_api_host_ifc_trap_id_register_set(api_handle, SX_ACCESS_CMD_REGISTER, DEFAULT_ETH_SWID, SX_TRAP_ID_PUDE, &port_channel))) {
goto out;
}
...
for (uint32_t ii = 0; ii < (sizeof(mlnx_trap_ids) / sizeof(*mlnx_trap_ids)); ii++) {
status = sx_api_host_ifc_trap_id_register_set(api_handle, SX_ACCESS_CMD_REGISTER, DEFAULT_ETH_SWID, mlnx_trap_ids[ii], &callback_channel);
}
while (!event_thread_asked_to_stop) {
FD_ZERO(&descr_set);
FD_SET(port_channel.channel.fd.fd, &descr_set);
FD_SET(callback_channel.channel.fd.fd, &descr_set);
...
ret_val = select(FD_SETSIZE, &descr_set, NULL, NULL, &timeout);
if (ret_val > 0) {
// Port state change event
if (FD_ISSET(port_channel.channel.fd.fd, &descr_set)) {
// Parse port state event here ...
if (g_notification_callbacks.on_port_state_change) {
g_notification_callbacks.on_port_state_change(1, &port_data);
}
}
if (FD_ISSET(callback_channel.channel.fd.fd, &descr_set)) {
// Receive notification event.
packet_size = MAX_PACKET_SIZE;
if (SX_STATUS_SUCCESS != (status = sx_lib_host_ifc_recv(&callback_channel.channel.fd, p_packet, &packet_size, receive_info))) {
goto out;
}
// BFD packet event
if (SX_TRAP_ID_BFD_PACKET_EVENT == receive_info->trap_id) {
const struct bfd_packet_event *event = (const struct bfd_packet_event*)p_packet;
// Parse and check event valid here ...
status = mlnx_switch_bfd_packet_handle(event);
continue;
}
// Same way to handle BFD timeout event, Bulk counter ready event. Emiited.
// FDB event and packet event handling
if (receive_info->trap_id == SX_TRAP_ID_FDB_EVENT) {
trap_name = "FDB event";
} else if (SAI_STATUS_SUCCESS != (status = mlnx_translate_sdk_trap_to_sai(receive_info->trap_id, &trap_name, &trap_oid))) {
continue;
}
if (SX_TRAP_ID_FDB_EVENT == receive_info->trap_id) {
// Parse FDB events here ...
if (g_notification_callbacks.on_fdb_event) {
g_notification_callbacks.on_fdb_event(event_count, fdb_events);
}
continue;
}
// Packet event handling
status = mlnx_get_hostif_packet_data(receive_info, &attrs_num, callback_data);
if (g_notification_callbacks.on_packet_event) {
g_notification_callbacks.on_packet_event(switch_id, packet_size, p_packet, attrs_num, callback_data);
}
}
}
}
out:
...
}
Next, let's use the FDB event as an example. When ASIC receives an FDB event, it will be fetched by the event handling loop above and the g_notification_callbacks.on_fdb_event function will be called to handle it. This function then calls the NotificationHandler::onFdbEvent function that was set up when Syncd was initialized, which serializes the event and forwards it through the message queue to the notification handler thread for processing:
// File: src/sonic-sairedis/syncd/NotificationHandler.cpp
void NotificationHandler::onFdbEvent(_In_ uint32_t count, _In_ const sai_fdb_event_notification_data_t *data)
{
std::string s = sai_serialize_fdb_event_ntf(count, data);
enqueueNotification(SAI_SWITCH_NOTIFICATION_NAME_FDB_EVENT, s);
}
And then the notification handler thread is woken up, takes the event out of the message queue, and then gets a lock on Syncd via Syncd and starts processing the notification again: the
// File: src/sonic-sairedis/syncd/NotificationProcessor.cpp
void NotificationProcessor::ntf_process_function()
{
std::mutex ntf_mutex;
std::unique_lock<std::mutex> ulock(ntf_mutex);
while (m_runThread) {
// When notification arrives, it will signal this condition variable.
m_cv.wait(ulock);
// Process notifications in the queue.
swss::KeyOpFieldsValuesTuple item;
while (m_notificationQueue->tryDequeue(item)) {
processNotification(item);
}
}
}
// File: src/sonic-sairedis/syncd/Syncd.cpp
// Call from NotificationProcessor::processNotification
void Syncd::syncProcessNotification(_In_ const swss::KeyOpFieldsValuesTuple& item)
{
std::lock_guard<std::mutex> lock(m_mutex);
m_processor->syncProcessNotification(item);
}
The next step is the distribution and processing of events. The syncProcessNotification function is a series of if-else statements that, depending on the type of event, call different handler functions to process the event:
// File: src/sonic-sairedis/syncd/NotificationProcessor.cpp
void NotificationProcessor::syncProcessNotification( _In_ const swss::KeyOpFieldsValuesTuple& item)
{
std::string notification = kfvKey(item);
std::string data = kfvOp(item);
if (notification == SAI_SWITCH_NOTIFICATION_NAME_SWITCH_STATE_CHANGE) {
handle_switch_state_change(data);
} else if (notification == SAI_SWITCH_NOTIFICATION_NAME_FDB_EVENT) {
handle_fdb_event(data);
} else if ...
} else {
SWSS_LOG_ERROR("unknown notification: %s", notification.c_str());
}
}
And each event handling function is similar in that they deserialize the events sent to them and then call the real processing logic to send notifications, for example, the handle_fdb_event function and process_on_fdb_event corresponding to the fdb event:
// File: src/sonic-sairedis/syncd/NotificationProcessor.cpp
void NotificationProcessor::handle_fdb_event(_In_ const std::string &data)
{
uint32_t count;
sai_fdb_event_notification_data_t *fdbevent = NULL;
sai_deserialize_fdb_event_ntf(data, count, &fdbevent);
process_on_fdb_event(count, fdbevent);
sai_deserialize_free_fdb_event_ntf(count, fdbevent);
}
void NotificationProcessor::process_on_fdb_event( _In_ uint32_t count, _In_ sai_fdb_event_notification_data_t *data)
{
for (uint32_t i = 0; i < count; i++) {
sai_fdb_event_notification_data_t *fdb = &data[i];
// Check FDB event notification data here
fdb->fdb_entry.switch_id = m_translator->translateRidToVid(fdb->fdb_entry.switch_id, SAI_NULL_OBJECT_ID);
fdb->fdb_entry.bv_id = m_translator->translateRidToVid(fdb->fdb_entry.bv_id, fdb->fdb_entry.switch_id, true);
m_translator->translateRidToVid(SAI_OBJECT_TYPE_FDB_ENTRY, fdb->fdb_entry.switch_id, fdb->attr_count, fdb->attr, true);
...
}
// Send notification
std::string s = sai_serialize_fdb_event_ntf(count, data);
sendNotification(SAI_SWITCH_NOTIFICATION_NAME_FDB_EVENT, s);
}
The logic for sending specific events is pretty straightforward, and ultimately it's all about sending notifications to ASIC_DB via [NotificationProducer](. /4-2-2-redis-messaging-layer.html#notificationproducer--notificationconsumer) to send notifications to the ASIC_DB:
// File: src/sonic-sairedis/syncd/NotificationProcessor.cpp
void NotificationProcessor::sendNotification(_In_ const std::string& op, _In_ const std::string& data)
{
std::vector<swss::FieldValueTuple> entry;
sendNotification(op, data, entry);
}
void NotificationProcessor::sendNotification(_In_ const std::string& op, _In_ const std::string& data, _In_ std::vector<swss::FieldValueTuple> entry)
{
m_notifications->send(op, data, entry);
}
// File: src/sonic-sairedis/syncd/RedisNotificationProducer.cpp
void RedisNotificationProducer::send(_In_ const std::string& op, _In_ const std::string& data, _In_ const std::vector<swss::FieldValueTuple>& values)
{
std::vector<swss::FieldValueTuple> vals = values;
// The m_notificationProducer is created in the ctor of RedisNotificationProducer as below:
// m_notificationProducer = std::make_shared<swss::NotificationProducer>(m_db.get(), REDIS_TABLE_NOTIFICATIONS_PER_DB(dbName));
m_notificationProducer->send(op, data, vals);
}
At this point, the process of notification upload in Syncd is finished.