The regularity of devastating cyber-attacks has made cybersecurity a grand societal challenge. To combat this societal issue, many organizations have aimed to develop timely, relevant, and actionable intelligence about emerging threats and key threat actors to enable effective cybersecurity decisions. This process, also referred to as Cyber Threat Intelligence (CTI), has quickly emerged as a key aspect of cybersecurity. At its core, CTI is a data-driven process that relies on the systematic and large-scale analysis of log files, malware binaries, events, Open Source Intelligence (OSINT), and other rapidly evolving cybersecurity data sources. Artificial intelligence (AI)-based methods such as machine learning, data mining, text mining, network science, and deep learning hold significant promise in sifting through large quantities of structured, unstructured, and semi-structured cybersecurity data to deliver novel CTI capabilities with unprecedented efficiency and effectiveness. Despite their rapid proliferation through the academic and industry CTI landscape, AI methods are often black boxes. As a result, it is often unclear how and/or why an algorithm executed its decision-making process. Lack of interpretability can affect model performance, prevent systematic model tuning, and reduce algorithm trustworthiness. Ultimately, these drawbacks hinder key stakeholders (e.g., security analysts) from effectively leveraging AI-based decisions for critical CTI tasks (e.g., security control deployment).