TA18-276B: Advanced Persistent Threat Activity Exploiting Managed Service Providers

["US-CERT" <US-CERT () ncas us-cert gov>]

U.S. Department of Homeland Security US-CERT

National Cyber Awareness System:



TA18-276B: Advanced Persistent Threat Activity Exploiting Managed Service Providers [ 
https://www.us-cert.gov/ncas/alerts/TA18-276B ] 10/03/2018 07:47 AM EDT 
Original release date: October 03, 2018

Systems Affected

Network Systems

Overview

The National Cybersecurity and Communications Integration Center (NCCIC) is aware of ongoing APT actor activity 
attempting to infiltrate the networks of global managed service providers (MSPs). Since May 2016, APT actors have used 
various tactics, techniques, and procedures (TTPs) for the purposes of cyber espionage and intellectual property theft. 
APT actors have targeted victims in several U.S. critical infrastructure sectors, including Information Technology 
(IT), Energy, Healthcare and Public Health, Communications, and Critical Manufacturing.

This Technical Alert (TA) provides information and guidance to assist MSP customer network and system administrators 
with the detection of malicious activity on their networks and systems and the mitigation of associated risks. This TA 
includes an overview of TTPs used by APT actors in MSP network environments, recommended mitigation techniques, and 
information on reporting incidents.

Description

MSPs provide remote management of customer IT and end-user systems. The number of organizations using MSPs has grown 
significantly over recent years because MSPs allow their customers to scale and support their network environments at a 
lower cost than financing these resources internally. MSPs generally have direct and unfettered access to their 
customers networks, and may store customer data on their own internal infrastructure. By servicing a large number of 
customers, MSPs can achieve significant economies of scale. However, a compromise in one part of an MSPs network can 
spread globally, affecting other customers and introducing risk.

Using an MSP significantly increases an organizations virtual enterprise infrastructure footprint and its number of 
privileged accounts, creating a larger attack surface for cyber criminals and nation-state actors. By using compromised 
legitimate MSP credentials (e.g., administration, domain, user), APT actors can move bidirectionally between an MSP and 
its customers shared networks. Bidirectional movement between networks allows APT actors to easily obfuscate detection 
measures and maintain a presence on victims networks.

*Note:* NCCIC previously released information related to this activity in Alert TA17-117A: Intrusions Affecting 
Multiple Victims Across Multiple Sectors [ https://www.us-cert.gov/ncas/alerts/TA17-117A ] published on April 27, 2017, 
which includes indicators of compromise, signatures, suggested detection methods, and recommended mitigation techniques.

Technical Details

APT

APT actors use a range of living off the land techniques to maintain anonymity while conducting their attacks. These 
techniques include using legitimate credentials and trusted off-the-shelf applications and pre-installed system tools 
present in MSP customer networks.

Pre-installed system tools, such as command line scripts, are very common and used by system administrators for 
legitimate processes. Command line scripts are used to discover accounts and remote systems.

PowerSploit is a repository of Microsoft PowerShell and Visual Basic scripts and uses system commands such as netsh. 
PowerSploit, originally developed as a legitimate penetration testing tool, is widely misused by APT actors. These 
scripts often cannot be blocked because they are legitimate tools, so APT actors can use them and remain undetected on 
victim networks. Although network defenders can generate log files, APT actors use of legitimate scripts makes it 
difficult to identify system anomalies and other malicious activity.

When APT actors use system tools and common cloud services, it can also be difficult for network defenders to detect 
data exfiltration. APT actors have been observed using Robocopya Microsoft command line toolto transfer exfiltrated and 
archived data from MSP client networks back through MSP network environments. Additionally, APT actors have been 
observed using legitimate PuTTY Secure Copy Client functions, allowing them to transfer stolen data securely and 
directly to third-party systems.

Impact

A successful network intrusion can have severe impacts to the affected organization, particularly if the compromise 
becomes public. Possible impacts include


  * Temporary or permanent loss of sensitive or proprietary information, 
  * Disruption to regular operations, 
  * Financial losses to restore systems and files, and 
  * Potential harm to the organizations reputation. 

Solution

Detection

Organizations should configure system logs to detect incidents and to identify the type and scope of malicious 
activity. Properly configured logs enable rapid containment and appropriate response.

Response

An organizations ability to rapidly respond to and recover from an incident begins with the development of an incident 
response capability. An organizations response capability should focus on being prepared to handle the most common 
attack vectors (e.g., spearphishing, malicious web content, credential theft). In general, organizations should prepare 
by


  * Establishing and periodically updating an incident response plan. 
  * Establishing written guidelines that prioritize incidents based on mission impact, so that an appropriate response 
can be initiated. 
  * Developing procedures and out-of-band lines of communication to handle incident reporting for internal and external 
relationships. 
  * Exercising incident response measures for various intrusion scenarios regularly, as part of a training regime. 
  * Committing to an effort that secures the endpoint and network infrastructure: prevention is less costly and more 
effective than reacting after an incident. 

Mitigation

Manage Supply Chain Risk

MSP clients that do not conduct the majority of their own network defense should work with their MSP to determine what 
they can expect in terms of security. MSP clients should understand the supply chain risk associated with their MSP. 
Organizations should manage risk equally across their security, legal, and procurement groups. MSP clients should also 
refer to cloud security guidance from the National Institute of Standards and Technology to learn about MSP terms of 
service, architecture, security controls, and risks associated with cloud computing and data protection.[1] [ 
https://www.nist.gov/itl/nist-cloud-computing-related-publications ] [2] [ 
https://bigdatawg.nist.gov/_uploadfiles/M0008_v1_7256814129.pdf ] [3] [ 
https://csrc.nist.gov/publications/detail/sp/800-144/final ]

Architecture

Restricting access to networks and systems is critical to containing an APT actors movement. Provided below are key 
items that organizations should implement and periodically audit to ensure their network environments physical and 
logical architecture limits an APT actors visibility and access.

"Virtual Private Network Connection Recommendations"


  * *Use a dedicated Virtual Private Network (VPN) for MSP connection. *The organizations local network should connect 
to the MSP via a dedicated VPN. The VPN should use certificate-based authentication and be hosted on its own device. 
  * *Terminate VPN within a demilitarized zone (DMZ).* The VPN should terminate within a DMZ that is isolated from the 
internal network. Physical systems used within the DMZ should not be used on or for the internal network. 
  * *Restrict VPN traffic to and from MSP. *Access to and from the VPN should be confined to only those networks and 
protocols needed for service. All other internal networks and protocols should be blocked. At a minimum, all failed 
attempts should be logged. 
  * *Update VPN authentication certificates annually.* Update the certificates used to establish the VPN connection no 
less than annually. Consider rotating VPN authentication certificates every six months. 
  * *Ensure VPN connections are logged, centrally managed, and reviewed.* All VPN connection attempts should be logged 
in a central location. Investigate connections using dedicated certificates to confirm they are legitimate. 

"Network Architecture Recommendations"


  * *Ensure internet-facing networks reside on separate physical systems.* All internet-accessible network zones (e.g., 
perimeter network, DMZ) should reside on their own physical systems, including the security devices used to protect the 
network environment. 
  * *Separate internal networks by function, location, and risk profile.* Internal networks should be segmented by 
function, location, and/or enterprise workgroup. All communication between networks should use Access Control Lists and 
security groups to implement restrictions. 
  * *Use firewalls to protect server(s) and designated high-risk networks. *Firewalls should reside at the perimeter of 
high-risk networks, including those hosting servers. Access to these networks should be properly restricted. 
Organizations should enable logging, using a centrally managed logging system. 
  * *Configure and enable private Virtual Local Area Networks (VLANs).* Enable private VLANs and group them according 
to system function or user workgroup. 
  * *Implement host firewalls. *In addition to the physical firewalls in place at network boundaries, hosts should also 
be equipped and configured with host-level firewalls to restrict communications from other workstations (this decreases 
workstation-to-workstation communication). 

"Network Service Restriction Recommendations"


  * *Only permit authorized network services outbound from the internal network.* Restrict outbound network traffic to 
only well-known web browsing services (e.g., Transmission Control Protocol [TCP]/80, TCP/443). In addition, monitor 
outbound traffic to ensure the ports associated with encrypted traffic are not sending unencrypted traffic. 
  * *Ensure internal and external Domain Name System (DNS) queries are performed by dedicated servers. *All systems 
should leverage dedicated internal DNS servers for their queries. Ensure that DNS queries for external hosts using User 
Datagram Protocol (UDP)/53 are permitted for only these hosts and are filtered through a DNS reputation service, and 
that outbound UDP/53 network traffic by all other systems is denied. Ensure that TCP/53 is not permitted by any system 
within the network environment. All attempts to use TCP/53 and UDP/53 should be centrally logged and investigated. 
  * *Restrict access to unauthorized public file shares. *Access to public file shares that are not used by the 
organizationsuch as Dropbox, Google Drive, and OneDriveshould be denied. Attempts to access public file share sites 
should be centrally logged and investigated. Recommended additional action: monitor all egress traffic for possible 
exfiltration of data. 
  * *Disable or block all network services that are not required at network boundary.* Only those services needed to 
operate should be enabled and/or authorized at network boundaries. These services are typically limited to TCP/137, 
TCP/139, and TCP/445. Additional services may be needed, depending on the network environment, these should be tightly 
controlled to only send and receive from certain whitelisted Internet Protocol addresses, if possible. 

Authentication, Authorization, and Accounting

Compromised account credentials continue to be the number one way threat actors are able to penetrate a network 
environment. The accounts organizations create for MSPs increase the risk of credential compromise, as MSP accounts 
typically require elevated access. It is important organizations adhere to best practices for password and permission 
management, as this can severely limit a threat actors ability to access and move laterally across a network. Provided 
below are key items organizations should implement and routinely audit to ensure these risks are mitigated.

"Account Configuration Recommendations"


  * *Ensure MSP accounts are not assigned to administrator groups. *MSP accounts should not be assigned to the 
Enterprise Administrator (EA) or Domain Administrator (DA) groups. 
  * *Restrict MSP accounts to only the systems they manage. *Place systems in security groups and only grant MSP 
account access as required. Administrator access to these systems should be avoided when possible. 
  * *Ensure MSP account passwords adhere to organizational policies. *Organizational password policies should be 
applied to MSP accounts. These policies include complexity, life, lockout, and logging. 
  * *Use service accounts for MSP agents and services. *If an MSP requires the installation of an agent or other local 
service, create service accounts for this purpose. Disable interactive logon for these accounts. 
  * *Restrict MSP accounts by time and/or date. *Set expiration dates reflecting the end of the contract on accounts 
used by MSPs when those accounts are created or renewed. Additionally, if MSP services are only required during 
business hours, time restrictions should also be enabled and set accordingly. Consider keeping MSP accounts disabled 
until they are needed and disabling them once the work is completed. 
  * *Use a network architecture that includes account tiering.* By using an account tiering structure, higher 
privileged accounts will never have access or be found on lower privileged layers of the network. This keeps EA and DA 
level accounts on the higher, more protected tiers of the network. Ensure that EA and DA accounts are removed from 
local administrator groups on workstations. 

"Logging Configuration Recommendations"


  * *Enable logging on all network systems and devices and send logs to a central location.* All network systems and 
devices should have their logging features enabled. Logs should be stored both locally and centrally to ensure they are 
preserved in the event of a network failure. Logs should also be backed up regularly and stored in a safe location. 
  * *Ensure central log servers reside in an enclave separate from other servers and workstations. *Log servers should 
be isolated from the internet and network environment to further protect them from compromise. The firewall at the 
internal network boundary should only permit necessary services (e.g., UDP/514). 
  * *Configure local logs to store no less than seven days of log data. *The default threshold for local logging is 
typically three days or a certain file size (e.g., 5 MB). Configure local logs to store no less than seven days of log 
data. Seven days of logs will cover the additional time in which problems may not be identified, such as holidays. In 
the event that only size thresholds are available, NCCIC recommends that this parameter be set to a large value (e.g., 
512MB to1024MB) to ensure that events requiring a high amount of log data, such as brute force attacks, can be 
adequately captured. 
  * *Configure central logs to store no less than one year of log data.* Central log servers should store no less than 
a years worth of data prior to being rolled off. Consider increasing this capacity to two years, if possible. 
  * *Install and properly configure a Security Information and Event Management (SIEM) appliance.* Install a SIEM 
appliance within the log server enclave. Configure the SIEM appliance to alert on anomalous activity identified by 
specific events and on significant derivations from baselined activity. 
  * *Enable PowerShell logging. *Organizations that use Microsoft PowerShell should ensure it is upgraded the latest 
version (minimum version 5) to use the added security of advanced logging and to ensure these logs are being captured 
and analyzed. PowerShells features include advanced logging, interaction with application whitelisting (if using 
Microsofts AppLocker), constrained language mode, and advanced malicious detection with Antimalware Scan Interface. 
These features will help protect an organizations network by limiting what scripts can be run, logging all executed 
commands, and scanning all scripts for known malicious behaviors. 
  * *Establish and implement a log review process. *Logs that go unanalyzed are useless. It is critical to network 
defense that organizations establish a regular cycle for reviewing logs and developing analytics to identify patterns. 

Operational Controls

Building a sound architecture supported by strong technical controls is only the first part to protecting a network 
environment. It is just as critical that organizations continuously monitor their systems, update configurations to 
reflect changes in their network environment, and maintain relationships with MSPs. Listed below are key operational 
controls organizations should incorporate for protection from threats.

"Operational Control Recommendations"


  * *Create a baseline for system and network behavior. *System, network, and account behavior should be baselined to 
make it easier to track anomalies within the collected logs. Without this baseline, network administrators will not be 
able to identify the normal behaviors for systems, network traffic, and accounts. 
  * *Review network device configurations every six months. *No less than every six months, review the active 
configurations of network devices for unauthorized settings (consider reviewing more frequently). Baseline 
configurations and their checksums should be stored in a secure location and be used to validate files. 
  * *Review network environment Group Policy Objects (GPOs) every six months.* No less than every six months, review 
GPOs for unauthorized settings (consider reviewing more frequently). Baseline configurations and their checksums should 
be stored in a secure location and be used to validate files. 
  * *Continuously monitor and investigate SIEM appliance alerts. *The SIEM appliance should be continuously monitored 
for alerts. All events should be investigated and documented for future reference. 
  * *Periodically review SIEM alert thresholds. *Review SIEM appliance alert thresholds no less than every three 
months. Thresholds should be updated to reflect changes, such as new systems, activity variations, and new or old 
services being used within the network environment. 
  * *Review privileged account groups weekly. *Review privileged account groupssuch as DAs and EAsno less than weekly 
to identify any unauthorized modifications. Consider implementing automated monitoring for these groups. 
  * *Disable or remove inactive accounts. *Periodically monitor accounts for activity and disable or remove accounts 
that have not been active within a certain period, not to exceed 30 days. Consider including account management into 
the employee onboarding and offboarding processes. 
  * *Regularly update software and operating systems. *Ensuring that operating systems and software is up-to-date is 
critical for taking advantage of a vendors latest security offerings. These offerings can include mitigating known 
vulnerabilities and offering new protections (e.g., credential protections, increased logging, forcing signed 
software). 

It is important to note thatwhile the recommendations provided in this TA aim at preventing the initial attack vectors 
and the spread of any malicious activitythere is no single solution to protecting and defending a network. NCCIC 
recommends network defenders use a defense-in-depth strategy to increase the odds of successfully identifying an 
intrusion, stopping malware, and disrupting threat actor activity. The goal is to make it as difficult as possible for 
an attacker to be successful and to force them to use methods that are easier to detect with higher operational costs.

Report Unauthorized Network Access

*Contact DHS or your local FBI office immediately.* To report an intrusion and request resources for incident response 
or technical assistance, contact NCCIC at (NCCICCustomerService () hq dhs gov [ 
https://www.us-cert.govmailto:NCCICCustomerService () hq dhs gov ] or 888-282-0870), FBI through a local field office, 
or the FBIs Cyber Division (CyWatch () fbi gov [ https://www.us-cert.govmailto:CyWatch () fbi gov ] or 855-292-3937).

References

  * NIST Cloud Computing-Related Publications [ https://www.nist.gov/itl/nist-cloud-computing-related-publications ] 
  * NIST SP 500-292: Cloud Computing Reference Architecture [ 
https://bigdatawg.nist.gov/_uploadfiles/M0008_v1_7256814129.pdf ] 
  * NIST SP 800-144: Guidelines on Security and Privacy in Public Cloud Computing [ 
https://csrc.nist.gov/publications/detail/sp/800-144/final ] 

Revision History

  * October, 3 2018: Initial version 
________________________________________________________________________

This product is provided subject to this Notification [ http://www.us-cert.gov/privacy/notification ] and this Privacy 
& Use [ http://www.us-cert.gov/privacy/ ] policy.

body { font-size: 1em; font-family: Arial, Verdana, sans-serif; font-weight: normal; font-style: normal; color: 
#333333; } ________________________________________________________________________

A copy of this publication is available at www.us-cert.gov [ https://www.us-cert.gov ]. If you need help or have 
questions, please send an email to info () us-cert gov. Do not reply to this message since this email was sent from a 
notification-only address that is not monitored. To ensure you receive future US-CERT products, please add US-CERT () 
ncas us-cert gov to your address book. 

OTHER RESOURCES: Contact Us [ http://www.us-cert.gov/contact-us/ ] | Security Publications [ 
http://www.us-cert.gov/security-publications ] | Alerts and Tips [ http://www.us-cert.gov/ncas ] | Related Resources [ 
http://www.us-cert.gov/related-resources ]  

STAY CONNECTED: Sign up for email updates [ http://public.govdelivery.com/accounts/USDHSUSCERT/subscriber/new ]