Penn Computing

Penn Computing

Computing Menu Computing A-Z
Computing Home Information Systems & Computing Penn

ISC Network Services Vendor Escalation

ISC Networking has a dependency on outside vendors for some of our major services:

Following are recommended steps to take should we have a service failure for a vendor supplied service. In any situation where the Oncall operator isn't sure how or doesn't have the authority to accomplish the recommended steps, the Oncall operator can stop and get a Systems person and/or the primary contact for the service involved. The Oncall operator can continue to help and the Oncall operator together with the staff member to whom this was escalated should determine who will take responsibility for the remaining steps.

Problems with Akamai caching

  1. Determine if the problem is with Akamai or with our own server.


  2. Turn off Akamai caching.
    If you have determined that Akamai is having a server-wide problem, you may need to turn off Akamai caching. To turn off the Akamai caching:
    • change the DNS record.
      www.upenn.edu is a CNAME for www.upenn.edu.edgesuite.net. Change the DNS record so that www.upenn.edu is a CNAME for origin.www.upenn.edu. If you don't have authority to make this change, contact the NOC and ask them to make that change.
    • If you can't wait for the DNS data to propagate to the noc servers, which can take up to an hour regardless of the TTLs on the old CNAME records, flush the DNS cache on all 3 noc servers
      sudo -u named rndc flush
      We want to avoid flushing the cache, if possible. Use this command judiciously.


  3. Check out the administrative interface.
    The Akamai administrative interface is available at https://control.akamai.com/. You will need a username/password to access this site. Once logged in, you may get an online advisory from Akamai notifying you of any outages and you can bring up the monitor tool to see graphs of current activity. You can also open a trouble ticket by following the online links:
    Support (in the toolbar) ->
    Open a Ticket (in the Issue Tracking section) ->
    Open New Ticket
  4. You can get to the "Open a New Ticket" page directly by going to https://control.akamai.com/portal/cfit/open_ticket.jsp. If you haven't already logged in, you'll be asked for a username/password.

  5. Call the toll-free technical support number.
    If this is an emergency or you can't get to the Akamai administrative interface, call the toll-free number:
    1-877-4AKATEC
    Our account manager is Sandeep Chouganjkar, schougan@akamai.com, 617-714-6431. Once you open a ticket, Sandeep will be notified and she has contacted us within minutes of opening a ticket.

  6. Send out notification.
    • If the person primarily responsible for the service hasn't been contacted yet, contact that person.
    • Contact the ProDesk by phone. The ProDesk should already be available in the OnCall phone's phone book.
      215-573-4017 - voice
      11106 - Nextel radio
    • Prepare an outage message that goes to the NOC for delivery through the campus outage notification mechanism.


Problems with Messagelabs

Tower outage
  1. Determine which tower(s) is/are affected
    • mail42.messagelabs.com
    • mail46.messagelabs.com
  2. Check out the administrative interface.
    The Messagelabs administrative interface is available at https://clients.messagelabs.com/. Username/password can found on pobox in: /usr/local/etc/virus-report/. Once logged in you can check for advisories from messagelabs.
  3. Call Messagelabs support: 1 (866) 807-6047. Ask for Landon Evans if the problem isn't being handled.

AOL Blacklisting
  1. Determine which tower(s) is/are affected:
    There's a python script "relaytest" that will permit choosing of arbitrary From/To/Relay host:
    i.e.
    # /usr/local/lib/virus-report/relaytest 
    From: [sender address]
    To: [recipient address]
    Relay: mail46.messagelabs.com
    Subject: [testing, please ignore]: This is a test
    Message: [testing, please ignore]: This is only a test
    
  2. Change outbound relay to use "good" messagelabs server.
  3. Contact AOL: http://postmaster.aol.com/servers/contact.html
  4. Call Messagelabs support: 1 (866) 807-6047.
Sending out notfication
  • If the person primarily responsible for the service has not been contacted yet, contact that person.
  • Contact the ProDesk by phone. The ProDesk should already be available in the OnCall phone's phone book.
    215-573-4017 - voice
    11106 - Nextel radio
  • Prepare an outage message that goes to pobox-sig to notify account administrators. Please note that only members subscribed to pobox-sig can send messages to the Listserv list pobox-sig. Most of us are not specifically subscribed so you'll need to send the message as engineering [at] isc.upenn.edu


Microsoft Premier Support

  • By Phone (Always use the phone for urgent and critical situations)
    1. Call 800-936-3100.
    2. You will be prompted to choose which product group you are calling in for support on. There are five choices: 1, 2, 3, 4 & 5.
    3. The "Customer Support Representative" (CSR) will ask you if this is a "New" or "Existing" case and for your name and access ID.
    4. The CSR will ask you for a description of the problem and which software you are running.
    5. Be sure to tell the CSR the severity of the problem:  1=Catastrophic, A=Critical, B=Urgent or C=Important.
    6. The CSR will go through a scripted set of questions to determine which support queue to transfer you to.
    7. The CSR will give you a case number: SRX############.
    8. You will be transferred to the appropriate support queue.
  • Web via Premier Online (Initial response time is typically 4 to 12 hours)
    1. Go to https://premier.microsoft.com
    2. Click "Submit Incident".
    3. Choose the product from the drop down list that you would like support on then click "Next".
    4. Complete the short form describing the question/issue and severity then click "Submit".
    5. You will be provided with a case number on-screen: SRZ############.
    6. You can check this web site for status updates to your case or you can optionally have updates sent to you via e-mail.

    Incident Severity Descriptions:

    1 – Catastrophic- Immediate response goal

    • System, network, server or critical application down catastrophically impacting production and/or profitability
    • Major negative business impact. It is catastrophic in both scope and exposure

    A – Critical – Immediate to 1 hour response goal

    • High-impact problem in which production, operations, or development are severely impacted; or where production and/or profitability will be severely impacted within days
    • Any issue that has a serious negative business impact

    B – Urgent – 2 hour response goal

    • Significant problem where production is proceeding, but in a significantly impaired fashion
    • Time sensitive issue important to long-term productivity, but is not causing an immediate work stoppage

    C – Important – 4 hour response goal

    • Important issue, but does not have significant current productivity impact - most common level

     

    Zimbra Support

    • By Web
      1. Check whether existing problem report exists by searching: http://bugzilla.zimbra.com/query.cgi
      2. Go to: https://support.zimbra.com/
      3. Click on "Submit A New Case"
      4. Open ticket (reference Bug ID# if case already exists)
    • By Phone
      The Zimbra Technical Support Team is available by telephone for customers who have Premium Support. Telephone support is available by calling +1 877-522-6334 during normal business hours Monday through Friday 9:00 am - 6:00 pm U.S. Pacific time (-8 GMT), excluding U.S. holidays.

    Dimension Data (Cloud Control application for O365 account management)

    • By Email
      1. Send mail to support@ceryx.com
      2. The sender will receive a response with a case number.
      3. Any replies back to the automated message will be automatically threaded back into the ticket.
      4. And how do we see the status of the case???
    • By Phone
      1. Call 877-667-6245 24x7 for high priority issues
top

Information Systems and Computing
University of Pennsylvania
Comments & Questions


Penn Computing University of Pennsylvania
Information Systems and Computing, University of Pennsylvania